Measuring safety climate in health care

Abstract

Aim

To review quantitative studies of safety climate in health care to examine the psychometric properties of the questionnaires designed to measure this construct.

Method

A systematic literature review was undertaken to study sample and questionnaire design characteristics (source, no of items, scale type), construct validity (content validity, factor structure and internal reliability, concurrent validity), within group agreement, and level of analysis.

Results

Twelve studies were examined. There was a lack of explicit theoretical underpinning for most questionnaires and some instruments did not report standard psychometric criteria. Where this information was available, several questionnaires appeared to have limitations.

Conclusions

More consideration should be given to psychometric factors in the design of healthcare safety climate instruments, especially as these are beginning to be used in large scale surveys across healthcare organisations.

In response to growing concern about patient safety, the Department of Health in the UK¹ and the Institute of Medicine in the USA² advised that healthcare organisations should consider adopting the safety management techniques used in other industries. The UK industrial safety regulator, the Health and Safety Executive, recommends that organisations operating in high risk industries should regularly assess their safety culture.³ Safety culture is “the product of individual and group values, attitudes, perceptions, competencies and patterns of behaviour that determine the commitment to, and the style and proficiency of, an organisation's safety management” (page 23).⁴ This is usually measured in industry by workforce questionnaire surveys to assess what is called the “safety climate”.⁵ Safety climate can be regarded as the surface features of the underlying safety culture.⁶ It assesses workforce perceptions of procedures and behaviours in their work environment that indicate the priority given to safety relative to other organisational goals.⁷,⁸,⁹ As organisations are inherently hierarchical in structure, there are multiple levels at which safety climate can be investigated—for example, individuals, work groups, departments, organizations.¹⁰ Safety climate data are generally collected at the individual level, then aggregated to a higher level. The degree of homogeneity of workers' perceptions, as a measure of climate strength, can also be considered.¹¹,¹²

A number of different instruments are used to measure safety climate in industry.⁶,¹³ The resulting data offer managers an additional perspective on the state of their safety management systems and can also be used for benchmarking purposes and trends analysis.¹⁴ It has been argued¹⁵,¹⁶ that the essential dimension is management commitment to safety: while this is probably fundamental, industrial researchers do measure other aspects. The most common⁶,¹³ are shown in box 1. In industry, workforce perceptions of safety climate have been linked to safety outcomes such as workforce injuries,¹⁷,¹⁸,¹⁹,²⁰,²¹ and to safety processes such as workers' behaviours.²²

Box 1 Safety climate features in industry and health care

Industry⁶,¹³

• Management/supervisors

• Safety systems

• Risks

• Work pressure

• Competence

• Procedures/rules

Health care

• Management/supervisors

• Safety systems

• Risk perception

• Job demands

• Reporting/speaking up

• Safety attitudes/behaviours

• Communication/feedback

• Teamwork

• Personal resources (e.g. stress)

• Organisational factors

Safety climate surveys are now being increasingly used in healthcare organisations²³ and several instruments have been developed. This paper reviews quantitative studies designed to investigate safety climate in health care, with particular attention devoted to the questionnaires. It provides a complementary analysis to a recent review of survey instruments,²⁴ with some overlap in the studies examined.

Methods

Four databases were searched: MEDLINE, PsychINFO, EBSCO, and Web of Science using the search terms “health care workers”, “hospital safety”, “patient safety”, “safety climate”, and “safety culture”. Relevant papers were retrieved and papers were also retrieved from patient safety conferences. A total of 29 studies were initially identified. The criteria for inclusion for detailed scrutiny were: (1) use of a questionnaire for individual response designed to measure safety climate or safety culture in a healthcare setting; (2) details provided of the measuring instrument; (3) tested on a sample of over 50 respondents; and (4) report published in English. From the 29 papers retrieved, 12 studies were identified as suitable for review.²⁵,²⁶,²⁷,²⁸,²⁹,³⁰,³¹,³²,³³,³⁴,³⁵,³⁶ Studies reporting different components of the same data set³⁷,³⁸,³⁹,⁴⁰,⁴¹ were only included once, and those that examined general organisational culture or climate variables (such as work pressure or role ambiguity) in relation to safety in health care⁴²,⁴³,⁴⁴ were not included. The analysis extracted information on the survey location and sample, safety climate measure, safety outcome variables, and the main findings. The specific psychometric properties⁴⁵,⁴⁶ considered (box 2) were the content validity, criterion validity, as well as the internal factor structure of the instrument.

Box 2 Psychometric criteria

• Content validity is the degree to which “elements of an assessment instrument are relevant to and representative of the targeted construct for a particular assessment purpose” (page 238).⁴⁷ Determination of whether the scales or item set of a safety climate questionnaire have good content validity can be made from a number of sources, e.g. relevant theory, empirical literature, expert judgement.

• Criterion related validity should be established by correlating the climate scores with outcome data, preferably collected by some method other than the questionnaire instrument. In the case of safety climate, these can be safety outcomes such as individual or organisational accident rates, or safety processes such as rates of behaviours that are deemed to be precursors of accidents (e.g. risk taking, rule breaking). In practice, industrial safety climate researchers frequently use self‐report measures of accident rates or unsafe behaviours collected on the questionnaire. This is not ideal because of common method bias.⁴⁸ However, there can be difficulties in accessing confidential accident data and, because questionnaires are completed anonymously (making it impossible to identify individual safety records), self‐report measures are sometimes the only means by which individual level criterion related validity can be established.

• Factor analysis reveals the underlying structure of a scale and shows whether there are distinct factors or themes being measured. It requires reasonably large data sets (of about 100) or a sample where there is a 10:1 ratio of participants to items.⁴⁶,⁴⁹ This ratio becomes less relevant for sample sizes above 300.⁵⁰ Factors with three items or fewer are too close to being variable specific and should be discarded.⁵¹,⁵² The internal reliability data for proposed/identified factors can also be assessed. A Cronbach's alpha score of 0.7 or higher is usually regarded as indicative of acceptable internal reliability.⁵³

The safety climate factors/dimensions given in each study (table 1) were independently categorised (with acceptable inter‐rater agreement) by two industrial psychologists (CB, SY) into 10 themes corresponding to distinct aspects of safety management. This was carried out by examining the content of the items loading on each subscale/factor where these were available. As most of the studies had based their definitions of safety climate on the literature from industry or had adapted industrial instruments, the themes were labelled in a similar fashion to those most commonly measured in industry (box 1).

Table 1 Studies measuring safety climate in health care.

Authors	Instrument details and survey sample	Safety climate factors	Psychometric analyses	Outcome measures	Results
(1) DeJoy et al26See also DeJoy et al,37,38 Gershon et al,39 Guastello et al,40 McGovern et al41 for analyses based on parts of same data set	Safety climate scale 35 items (part of a longer questionnaire)902 nurses, 322 physicians and 247 technicians (57% response rate) from 3 USA hospitals	Safety performance feedback Management commitment to safety Provision of PPE UP‐related job hindrances Knowledge and information about UP Risk of infection Self‐protective actions Work organisation	Exploratory factor analysis yielded 8 factors based on 23 items (some based on 3 items or less)α for the identified factors ranged from 0.39 (general work organisation) to 0.83 (management commitment)Regression analyses	Self‐report scale on compliance; adherence to UP	Job hindrance was strongest predictor of compliance in nurses and physicians. Safety performance feedback was a strong predictor of compliance in nurses but more modest predictor for physicians
(2) Gershon et al26	46 item safety climate scale (part of longer questionnaire)1240 employees from a large urban USA medical research centre (60% response rate)Only employees with the highest risk for blood and body fluid exposure were selected for participation	Senior management support Absence of job hindrances Cleanliness, orderliness Minimal conflict and good communication Safety‐related feedback/training by supervisors PPE/engineering control equipment availability	Exploratory factor analysis yielded 6 factors based on 20 items (only 2 based on more than 3 items)α > 0.7 for all 6 factors Regression analyses	Self‐report scale on compliance with universal safety precautions Exposure incident history	“Cleanliness and orderliness”, “senior management support” and “absence of job hindrances” associated with compliance with safety practices Higher “senior management support” and “feedback/training” related to lower exposure to incidents
(3) Neal et al27	16 items about safety climate (part of longer questionnaire)525 employees from an Australian hospital (56% response)	Safety climate scale included items about:Management values Communication Training Safety systems	Safety climate defined by mean score from 16 items No factor analysis α = 0.93 for 16 item safety climate scale Structural equation modelling	Self‐report of safety practices and procedural compliance	Safety climate indirectly related to safety compliance
(4) Felknor et al28	Safety climate 11 items based on Gershon et al.26 (part of longer quesionnaire)878 employees from 10 Costa Rican hospitals (96% response rate)	Management commitment to safety Work area Unsafe work practices Reporting safety violations	No FA57Regression analysis	Work injuries Self‐report compliance with safety practices	Safety climate inverse relationship with workplace injuries Positive relationship between safety climate and safety practices
(5) McCoy et al29	21 item safety climate scale based on studies by Murphy et al37149 infection control practitioners from 149 USA hospitals (62% response rate)	Management commitment Feedback Job demands Safety committee PPE availability	Exploratory FAα = 0.62–0.93 Logistic regression analysis	Perceptions of adequacy of healthcare worker training to monitor co‐workers' adherence to standard precautions	“Management commitment” and “feedback” positively related to training to observe co‐workers' standard precautions compliance“Job demands” inversely related to training to observe co‐workers' standard precautions compliance
(6) Vredenburgh30	18 item scale based on Ostrom et al5762 risk managers from 62 USA hospitals (57% response rate)	Rewards Training Management commitment Communication and feedback Selection Participation	Exploratory factor analysis; 6 factor solution did not correspond to the hypothesised dimensions Multiple regression	Occupational injuries	Factor 1 (reactive measures) and factor 2 (proactive measures) predicted injury rates
(7) Carrico31	79 item questionnaire based on Offshore Safety Questionnaire6393 nurses in Delaware, USA (31% response)	Communication Satisfaction safety Involvement Work pressure Safety attitudes Safety behaviours	Internal reliability analysis of proposed safety climate dimensions	None	Low mean scores indicated a somewhat poor safety climate for nurses
(8) Singer et al32	Stanford/PSCI Culture Survey (82 items)6312 employees including attending physicians, senior executives and a 10% random sample of other hospital personnel at 15 USA hospitals (47% response rate)	Organisation Department Production Reporting/seeking help Shame/self‐awareness	Exploratory FA yielded 5 factors based on 30 items (shame/self‐awareness factor only based on 3 items)α not given for identified factors	None	Problematic and neutral responses suggested “a lack of safety culture” in some hospitals
(9) Sorra and Nieva33	Hospital Survey on Patient Safety (79 items)1437 staff at 21 USA hospitals (29% response rate)	Supervisor/manager expectations and actions promoting patient safety Organisational learning Teamwork within units Communication openness Feedback/communication about error Response to error Staffing Hospital management support for patient safety Teamwork across hospital units Hospital handoffs and transitions	Exploratory FA yielded 14 factors based on 66 items. Confirmatory FA yielded a 12 factor solution (2 factors that measured outcomes and 10 factors that measured safety climate) based on 42 items (6 factors only based on 3 items)α>0.7 for all factors except staffing (α = 0.63)Correlations	Self‐report of:Number of events reported Overall patient safety grade Overall perceptions of safety Frequency of event reporting	“Overall perceptions of safety” were correlated with “patient safety grade” and “hospital management support for patient safety”“Frequency of event reporting” was correlated with “feedback and communication about error”
ORMAQ studies47,62
(10) Itoh et al34	Adapted Operating Team Resource Management Survey which included 57 items.66 doctors, 486 nurses and 43 pharmacists from 5 Japanese hospitals (91% response rate)	Satisfaction with management Morale and motivation Communication Teamwork Power distance Own competence Recognition of stress Stress management Error	No factor structure emerged from FA (personal communication)α not given for proposed factors Correlation	Rates of incident reporting for nurses in one hospital	“Recognition of human error” and “power distance” were negatively correlated with rates of incident reporting
(11) Pronovost et al35	Total length of questionnaire not given but it included 10 item safety climate scale 395 staff at a large USA teaching hospital (64% response rate)	Supervisor and management commitment to safety Knowledge of how to report adverse events Understanding of systems as the cause of adverse events	No FAα not given for proposed factors	None	Participants perceived supervisors to have a greater commitment to safety than senior leaders
(12) Woods et al36	Total length of questionnaire not given but it was adapted from the 60 item version of the ORMAQ 802 healthcare workers from an Australian Health Service Area (26% response rate)	Organisational culture Communication Teamwork Assertiveness Performance shaping factors Error	No FAα not given for proposed factors	None	None related to outcomes

PPE, personal protective equipment (e.g. gloves, masks); UP, universal precautions; FA, factor analysis; ORMAQ, Operating Room Management Attitudes Questionnaire.

Results

The 12 studies are described in table 1 in terms of the survey sample, instrument details (with any psychometric data), identified safety climate factors, outcome measures, and results. Most of the studies were from the USA and most sampled medical staff in different occupations (response rates 26–91%). The first nine studies used different safety climate measures while the other three used different versions of the Operating Room Management Attitudes Questionnaire (ORMAQ).⁵⁴ The ORMAQ was not originally designated as a general healthcare safety climate measure but recently it has been used for this purpose, so these studies were included for review. (Studies with the ORMAQ that did not specifically claim to be measuring organisational safety climate⁵⁵,⁵⁶ were excluded.)

Drawing from the information presented in table 1, the questionnaires were examined with respect to content validity, factor structure and internal reliability, and criterion related validity. The level of analysis used in the study is briefly considered below.

Content validity

Nine of the studies set out to measure safety climate, the remainder used the term safety culture, and one³⁰ used both terms. Four did not define either term. Definitions of safety climate were usually a version of shared perceptions of safety.¹⁰ A theoretical basis or model to outline proposed causal influences between safety climate and the safety outcome measures was rarely specified. One exception³² stated that their survey items were based on “elements of a culture of safety articulated by high reliability theory” (page 113). They listed seven components derived from this theory⁵⁸ but it is not clear how these relate to their questionnaire items and the five extracted factors did not provide confirmation. A circular model was given in another study²⁶ but it did not articulate any explanatory mechanism between safety climate and safety behaviours.

The 73 safety climate dimensions (table 1) were categorised into 10 safety management themes (box 1). Not surprisingly, given the origins of several of the measures, there was considerable overlap with the features measured in industry. Management commitment to safety emerged as the most frequently measured safety climate dimension in health care with nine studies including this. Three included supervisor commitment to safety. Safety systems—for example, personal protective equipment and safety training—were included in seven studies. Unlike the industrial sector, general attitudes to risk were not specifically addressed.

Work pressure is an important safety climate feature in industry and three healthcare instruments included a job demands/workload dimension. Unlike the industrial sector, competence did not emerge as a separate dimension, although two studies included measures about training and one included knowledge about universal precautions. Lastly, and in contrast to industry, compliance with procedures/rules did not emerge as a separate dimension in the healthcare measures, although two studies measured whether unsafe work practices were corrected by supervisors and workmates.

From the comparison it seems that at least three “core” dimensions from industry are being measured as components of safety climate in health care—management/supervisory commitment to safety, safety system, and work pressure. Sorra and Nieva³⁴ included two of these dimensions but did not assess perceptions of the safety system. Pronovost et al³⁶ also included two of these dimensions but their measure of the safety system was quite limited in scope as their safety climate scale only contained 10 items. Most of the other studies included one or two of these “core” dimensions each. This lends some weight to the argument for a set of universal or core variables that underpin safety climate across work sectors, although these probably need to be complemented with sets of sector specific factors.⁸,¹¹

Factor structure and internal reliability

For this review, studies with a sample size of less than 300 and factors consisting of three items or less were regarded with caution (box 2). Two studies conceptualised safety climate as a unidimensional construct but did not report a factor analysis to confirm this. Only six studies reported the results of a factor analysis. DeJoy et al²⁵ conducted separate exploratory factor analyses on their 35‐item measure of safety climate for each of the occupational groups studied (nurses, physicians, technicians). Each analysis yielded eight similar factors based on the same 23 items, as shown in table 1 (α coefficients ranged from 0.61 to 0.83 apart from general work organisation which was 0.39). However, more than half of these factors (feedback, knowledge and information, perceived risk, response efficacy, work organisation) were based on three items or less, which is usually regarded below minimal. Gershon et al²⁶ conducted an exploratory factor analysis on their 46‐item safety climate scale, yielding six factors (based on 20 items) that did not correspond to their nine hypothesised safety climate dimensions. An internal reliability analysis of the factor scales yielded acceptable α coefficients. Singer et al³² found a five‐factor solution that did not match their original thematic groupings. In two studies the data sets were rather small for the factor analyses that were conducted.²⁹,³⁰

Only one study provided a comprehensive report of scale development. For their Hospital Survey on Patient Safety, Sorra and Nieva³³ conducted an exploratory factor analysis to explore the dimensionality of their survey data. The results of the exploratory factor analysis revealed the existence of 14 distinct factors. A subsequent confirmatory factor analysis was conducted and the final confirmatory factor model contained 42 items in 12 factors (two factors which measured outcomes and 10 which measured safety climate). This model fitted the data well with good α coefficients.

Three studies used versions of the ORMAQ.³⁴,³⁵,³⁶ Despite sample sizes of more than 300, none of them reported a factor structure or internal reliabilities for the hypothesised dimensions. Itoh (personal communication, 2003) indicated that the results of an attempted factor analysis were not interpretable; a similar result has been reported elsewhere.⁵⁶

Criterion validity

Criterion or outcome measures of safety in health care could include worker behaviours, worker injuries, patient injuries, or other organisational outcomes (such as litigation costs). As shown in table 1, four studies had no outcome measures. Self‐reported worker rule compliance or event reporting behaviours were recorded in five studies, with two including independent measures of occupational injuries.²⁸,²⁹ Only one study had an independent measure of injuries or annoyances to patients,³⁴ although this was broadly conceptualised and included events such as losing artificial teeth. Other studies used the term “patient safety culture”³²,³⁵ but included no patient outcome measures.

The three studies using independent outcome measures reported significant associations between climate scores and outcomes. Two²⁸,³⁰ found evidence that positive perceptions of organisational safety in healthcare settings were related to fewer worker injuries. Itoh³⁴ measured the relationship between rates of nurses' reports of patient incidents from one of the participating hospitals from the preceding year and aggregated scores on the questionnaire. There was no correlation between questionnaire responses on incident reporting and rates of adverse events to patients. A significant negative correlation was reported between questionnaire scores on “recognition of inevitability of human error” and rates of incident reporting. This represents one of the few attempts to include an independent measure of worker safety behaviour but, because of the breadth of the measure and the small sample of units, these results would require to be interpreted with a degree of caution. Correlations of climate scores with self‐reported safety behaviours generally showed positive relationships.²⁵,²⁷,³³

Some care needs to be taken with the level of analysis for measuring and testing predictors against outcomes (that is, individual worker, team, department, hospital, NHS trust).⁵⁹,⁶⁰,⁶¹ Only one study³⁴ attempted to examine the relationship between safety climate of work units and their safety outcomes. Most aggregated questionnaire responses from the entire sample which often included more than one hospital.²⁵,²⁸,³²,³³ While Gershon et al²⁶ aggregated responses to the organisational level, their sample only included hospital employees who were at risk of exposure to blood and body fluids. Neal et al²⁷ and Pronovost et al³⁵ aggregated responses from representative samples of hospital workers to the organisational level. Although it can be argued that these studies produced more meaningful safety climate data, they did not examine the relationship with organisational safety outcomes such as worker or patient injury rates.

Discussion

The UK National Audit Office⁶⁵ has recently reported on the state of patient safety in NHS trusts. While offering an encouraging prognosis, this is far from a clean bill of health. The report states that “The safety culture within trusts is improving … However, trusts are still predominantly reactive in response to patient safety issues and parts of some organisations still operate a blame culture” (page 2). Measuring safety climate in health care helps to diagnose the underlying safety culture of an organisation or work unit. The prevailing culture influences safety behaviours and outcomes for both healthcare workers and patients. Safety climate questionnaires need to achieve as high a standard of measurement as possible so that healthcare managers can use the resulting data to design effective safety management systems and interventions.

We have reviewed the psychometric properties of instruments used to measure safety climate in 12 studies based in healthcare settings. None of these had achieved full scale testing and it is recognised that some instruments were at an early stage of development. The Hospital Survey on Patient Safety³³ met more of the specified psychometric criteria due to more systematic testing of internal structure than the other instruments reviewed. Some of the scales of this instrument—such as organisational learning/continuous improvement and teamwork (within, and across hospital units)—should not perhaps be considered part of the safety climate¹¹ unless their relationship with safety outcomes can be confirmed. This study only had a 29% response rate which was rather low compared with the other studies and may signal issues of usability or weaknesses in their survey method. Medical staff have limited time to complete and return questionnaires, so instruments for health care may need to be parsimonious and made available electronically as well as on paper to maximise response rates.

Several of the instruments had been developed from measures used in other industries (aviation, oil, nuclear). Considerable care needs to be taken when adapting measures from these very proceduralised high risk industries. Not only is the nature of the work very different, but the organisations have well defined hierarchical management structures with clear reporting relationships. Leadership issues are much more problematic to measure in health care as the managerial reporting relationships are subject to different interpretation by each professional group, thus introducing a degree of ambiguity. This is particularly true for doctors.⁶⁶ Moreover, the safety climate studies in industry all focus on worker injury rather than product (cf patient) damage. Determining reliable outcome measures for these healthcare studies appears to be challenging; sometimes the focus is on workers' behaviours, which might be regarded as safety process measures, and in other cases some kind of adverse event is used. As more patient safety indicator and outcome measures are being introduced, these should permit stronger data based on work unit and organisational performance to give appropriate higher level criterion metrics.

The data sets were drawn from different levels of organisational analysis and, as mentioned earlier, nested data of this type need to be analysed with some care—for example, by using multilevel modelling statistical techniques. Aggregating safety climate data across hospitals and, indeed, across healthcare systems in different countries⁶⁴ is not entirely advisable unless the questionnaire is measuring sector rather than organisational features. Zohar¹¹ has argued that safety climate can be meaningfully construed only at the group and organisational levels so as to reflect a particular supervisor's and senior management's influence on safety, respectively.

This is a preliminary review and it should be acknowledged that many of these research teams are now engaged in larger scale questionnaire studies. In future, meta‐analysis based on effect sizes will be needed to compare their results and to determine the validity and generalisability of the climate measures. In general, these studies have begun to confirm that safety climate scores can be associated with healthcare workers' safety behaviours or workers' injuries, replicating earlier findings from industry, although few independent measures were used. Very few of the reviewed studies considered the mechanisms that mediate the relationship between safety climate and safety outcomes (that is, worker injury or patient harm). In the wider literature on safety climate there are now models that attempt to explain the psychological mechanisms linking safety climate and worker behaviour.⁹,¹¹ In these models the relationships between safety climate, safety behaviour, and safety outcomes are focused on individual worker injury. In the healthcare sector there is an additional need to establish whether a different set of antecedents influences processes (worker behaviours) that affect patient safety outcomes as opposed to worker injury. In other words, are there different motivating factors that determine the safety behaviour of healthcare workers in relation to preventing personal harm compared with harm to a patient? The Institute of Medicine report¹ “To Err is Human” stated that “workers' safety is often linked with patient safety. If workers are safer in their jobs, patients will be safer also” (page 20). In fact there is little evidence to support this claim, although emerging evidence is encouraging.⁶⁷ So future safety climate research in health care should elaborate and test models that attempt to explicate the mechanisms influencing not only patient safety but worker safety as well.

Finally, while questionnaires offer an efficient and anonymous method of collecting safety climate data, researchers need to consider alternative techniques for sensing organisational culture. Ethnographic approaches based on observation and interviewing⁶⁸ can be expensive but they can provide valuable qualitative data to test the validity of the survey methods.