Abstract
Objective:
To create, administer, and psychometrically examine a survey to measure two new organizational culture factors – preoccupation with failure and adherence to shared baselines – in healthcare settings.
Method:
Direct care providers (n = 4484) from a large healthcare system in the Southern United States completed a survey as part of their annual safety culture assessment.
Results:
We provide evidence about the internal consistency (Cronbach’s alpha ranged from .80 to .89) factor structure, concurrent validity (correlation with overall patient safety grade ranged from .60 to .67, p <.05), and discriminant validity (correlations less than .85 with safety and teamwork culture) of these two factors.
Conclusions:
We established evidence for internal consistency and validity of two new factors that measure aspects of organizational culture – preoccupation with failure and adherence to shared baselines – that are distinct from safety culture and teamwork culture.
Keywords: Organisational learning, safety culture, high reliability, preoccupation with failure, adherence to shared baselines
Despite a renewed focus on patient safety since the Institute of Medicine report To Err is Human,1 there has been limited progress in ensuring that patients receive safer care in hospitals.2 Healthcare experts and researchers have argued that applying principles from high-reliability organizations (HROs) to health-care settings might help improve patient safety.3
At the core of high reliability organizing is a set of principles embodied in processes and practices that enable organizations to focus attention on emergent problems and to deploy the right set of resources to address those problems….High reliability organizing is characterized by five key principles that facilitate both problem detection and problem management.4
These five principles or factors are: (1) sensitivity to operations, (2) reluctance to simplify, (3) preoccupation with failure, (4) deference to expertise, and (5) resilience.5
Sammer et al.6 recently reviewed the patient safety culture literature and highlighted the processes that help organizations “achieve high reliability.” The authors identified three studies that focused on organizations with high reliability/zero defects,7–9 but none of these three articles measured high reliability explicitly. Only one study was identified via PubMed as measuring any constructs related to high reliability. In that study, Vogus et al. created the Safety Organizing Scale (SOS), a nine-item survey, that measures safety culture through the lens of the five HRO dimensions.10 While parsimonious, this instrument provides little depth within dimensions (i.e. too few items per dimension), making translation to interventions difficult. The Agency for Healthcare Research and Quality11 promotes steps organizations might take to achieve high reliability. However, there is a lack of practical tools to assess and improve high reliability behaviors.
Given the complexity and distinctiveness of each HRO factor, we decided to focus on measuring one of these factors – preoccupation with failure – along with a construct that we posit to be related to the general concept of HROs – adherence to shared baselines. We focused on adherence to shared baselines because reduction of variability and standardization of clinical work processes through the establishment and management of shared baselines has been a key ingredient of organizations that provide high-quality care (e.g. Intermountain Healthcare).12–14 Many current efforts to reduce patient safety risks attempt to increase the reliability and decrease the variability of care delivery through caregiver adherence to shared baselines. We reasoned that if adherence to shared baselines helps reduce variability and thereby increase reliability, the construct of adherence to shared baselines should be related to high reliability. In sum, our goal was to address an important research gap in the literature by (1) creating survey items to measure one traditional HRO factor– preoccupation with failure – along with one new and complementary factor – adherence to shared baselines and (2) assess the reliability and validity of these survey items.
Methods
Participants
We invited direct care providers from 13 hospitals within a large hospital system in the Southern United States to complete our survey. The hospitals included a large tertiary care teaching hospital, small community hospitals, and large urban hospitals. Of 10,927 direct care providers, 4484 completed the survey (41% response rate across the system). We used listwise deletion to handle missing data, resulting in a usable sample size of 3310 participants who provided answers to all survey items. IRB approval was obtained prior to initiating data collection. The survey was anonymous, administered via Survey Monkey and contained no incentives for participants. All direct care providers were sent an email with a link to the electronic survey twice during a six-week timeframe, and the link was also posted on the hospital system’s intranet website.
Measures
Survey items assessed safety culture, teamwork culture, overall patient safety grade, preoccupation with failure, adherence to shared baselines, and demographics. We used the safety culture and teamwork culture survey items from the Safety Attitudes Questionnaire15 so we could examine their associations with our two new factors. All culture survey items were measured on a five-point Likert-type scale, where 1 = disagree strongly and 5 = agree strongly. Participants were asked to grade their unit on patient safety from A = Excellent to E = Failing using the one-item measure from AHRQ’s Hospital Survey on Patient Safety Culture.16
We created survey items to measure preoccupation with failure and adherence to shared baselines. Given that this study was conducted as part of the system’s annual safety culture survey of all employees, we were limited in the amount of additional survey items we could include and therefore did not try to measure all of the HRO factors discussed by Weick and Sutcliffe. Additionally, we focused on these two factors because some of the other HRO factors appeared to be quite difficult to measure via survey methodology.
Three patient safety experts – a neonatologist (JP), a general internist (ET), and a psychometrician (JE) – jointly created four initial survey items to measure each of these two factors. The development process consisted of initially reviewing definitions from Weick and Sutcliffe5 and then iteratively developing items through face-to-face meetings and email to determine the best way to measure these constructs. We then asked 19 employees to pilot test these new items and provide input about anything that was unclear, confus-ing, or difficult to understand. Based on their com-ments, we revised the items.
Statistical analysis
We conducted a confirmatory factor analysis (CFA) instead of an exploratory factor analysis for our two new factors – preoccupation with failure and adherence to shared baselines – to provide evidence for the factor structure of our survey because we had a priori hypotheses about which items should correspond with which scales. The CFA results contain statistical and practical aspects of model fit,17 including the χ2/df ratio, Tucker-Lewis Index (TLI),18 comparative fit index (CFI),19 and the root mean square error of approximation (RMSEA).20 Model fit is considered good when χ2/df ratio is less than three, TLI and CFI values are greater than .90, and RMSEA values are .05 or less.21,22 We assessed internal consistency of each scale with Cronbach’s alpha, which should be greater than .70.
We then computed correlations between our two new culture factors and safety culture, teamwork culture, and overall patient safety grade at the unit level. We applied the Bonferroni correction when interpreting the significance of the correlational results to guard against Type I errors (i.e. finding an association when none exists). Our goal with these correlations was to estimate two types of validity – discriminant and concurrent – for the new factors. Discriminant validity allows us to have confidence that our factors of interest are not correlated too highly with other factors and therefore measure different content than those factors. Correlations less than .85 between factors are indicative of discriminant validity, which indicates that different constructs provide non-redundant information.23 We establish concurrent validity when our factors (i.e. pre-occupation with failure and adherence to shared baselines) are significantly related to an important outcome, such as overall patient safety grade.
Results
Study participants
Table 1 contains demographic information about study participants. More than three-quarters of the participants were women. Nurses were the most common participant, followed by patient care assistants and therapists. Seventy percent of the participants have worked in their specialty for five or more years and 80% work more than 70 h during each pay period.
Table 1.
Demographic information.
| Demographic variable | Frequency (%) |
|---|---|
| Gender | |
| Female | 2724 (86) |
| Male | 444 (14) |
| Missing | 142 |
| Position | |
| Nurse (LVN, RN) | 1874 (60) |
| Technologist/technician (e.g. Surg., Lab, Rad.) | 301 (10) |
| Patient care assistant | 245 (8) |
| Therapist (RT, PT, OT, other) | 204 (7) |
| Charge nurse/team leader | 141 (4) |
| Unit secretary | 58 (2) |
| Pharmacist | 47 (1) |
| Other | 264 (8) |
| Missing | 176 |
| Years in specialty | |
| 5 to 10 years | 842 (27) |
| 11 to 20 years | 811 (26) |
| 21 years or more | 529 (17) |
| 3 to 4 years | 372 (12) |
| 1 to 2 years | 327 (10) |
| Less than 6 months | 138 (4) |
| 6 to 11 months | 117 (4) |
| Missing | 174 |
| Hours per pay period | |
| More than 70 h each pay period | 2529 (80) |
| 40 to 70 h each pay period | 439 (14) |
| Less than 40 h each pay period | 181 (6) |
| Missing | 161 |
LVN: licensed vocational nurse; RN: registered nurse; RT: respiratory therapist; PT: physical therapist; OT: occupational therapist.
Descriptive statistics for survey items
Table 2 contains descriptive statistics for each of the items measuring high reliability culture. The mean values were generally higher for the items measuring preoccupation with failure (M ranged from 4.17 to 4.7) than the items for adherence to shared baselines (M ranged from 4.2 to 4.41). The first two items measuring preoccupation with failure had particularly high means (M1 = 4.70 and M2 = 4.66, respectively) and low standard deviations (sd1 = .65 and sd2 = .69, respectively), suggesting a possible ceiling effect.
Table 2.
High reliability culture items by dimension.
| Dimensions and items | Mean | Standard deviation |
|---|---|---|
| Preoccupation with failure | ||
| 1. We are trained to detect hazards that might harm a patient | 4.70 | .65 |
| 2. We are constantly on the lookout for hazards that might harm a patient | 4.66 | .69 |
| 3. Errors are uncommon because we work proactively to avoid mistakes before they occur | 4.39 | .89 |
| 4. After I report a patient safety concern, my unit and I receive feedback about actions taken to resolve it | 4.17 | 1.07 |
| Adherence to shared baselines | ||
| 1. I can usually predict the next steps in my patient’s care because healthcare team members adhere to mutually agreed upon care guidelines (i.e. national standards, bundles, pathways, standard operating procedures, etc.) | 4.41 | .89 |
| 2. We track if guidelines are followed | 4.30 | .96 |
| 3. Care guidelines are frequently revised based on clinician input | 4.20 | .98 |
| 4. When we deviate from existing guidelines we document why | 4.32 | .93 |
CFA results
The CFA results for both new culture factors are shown in Table 3. We first tested initial models separately for preoccupation with failure and adherence to shared baselines. Given that model fit was not perfect, we then examined suggested revisions possible to the model based on the results from the initial model. We examined a revised model for preoccupation with failure that allowed the first two items to covary because both items focused on similar content, namely “hazards.” We made a similar revision to the adherence to shared baselines model by allowing the second and fourth items to covary because they appeared to measure similar content (i.e. tracking/documenting guidelines). While the initial models provided mixed fit results for each of the high reliability culture factors, the revised models indicated significantly better fit. All revised models had improved fit from the initial models based on significant (p < .05) changes in Chi-square and improved RMSEA, CFI, and TLI values.
Table 3.
Confirmatory factor analysis results for new culture factors.
| Model | Type of model | χ2 | df | χ2/df ratio should be < 3 | Significance of Δ χ2 should be < .05 | RMSEA should be ≤ .05 | CFI should be ≥ .90 | TLI should be ≥ .90 |
|---|---|---|---|---|---|---|---|---|
| Pre-occupation with failure | Initial | 372.4 | 2 | 186.2 | – | .20 | .95 | .74 |
| Pre-occupation with failure | Revised | 4.2 | 1 | 4.2 | .001 | .03 | 1.0 | 1.0 |
| Adherence of shared baselines | Initial | 22.8 | 2 | 11.4 | – | .048 | 1.00 | .99 |
| Adherence of shared baselines | Revised | .09 | 1 | .09 | .001 | .000 | 1.00 | 1.00 |
RMSEA: root mean square error of approximation; CFI: comparative fit index; TLI: Tucker-Lewis Index.
Internal consistency and correlational results
Cronbach’s alpha estimates for all three high reliability culture principles were greater than .7, indicating good internal consistency (preoccupation with failure = .80; adherence to shared baselines = .89). Table 4 contains bivariate correlations between the different types of culture and overall patient safety grade at the unit level. All correlations among preoccupation with failure and adherence to shared baselines with safety culture and teamwork culture were less than .85 (ranging from r = .61, p < .05 to r = .75, p < .05), indicating support for discriminant validity. All types of culture were significantly correlated with overall patient safety grade, with preoccupation with failure being the strongest predictor (r = .67 and .66, p < .05, respectively), even stronger than safety culture (r = .63, p < .05) and teamwork culture (r = .61, p < .05)
Table 4.
Correlations between culture dimensions and overall patient safety grade (unit-level).
| Preoccupation with failure | Adherence to shared baselines | Safety culture | Teamwork culture | Overall patient safety grade | |
|---|---|---|---|---|---|
| Preoccupation with failure | – | ||||
| Adherence to shared baselines | .76* | – | |||
| Safety culture | .75* | .72* | – | ||
| Teamwork culture | .62* | .61* | .75* | – | |
| Overall patient safety grade | .67* | .60* | .63* | .61* | – |
Correlations significant at p <.05.
Scores by type of culture
Table 5 contains percent positive scores by type of culture for the different hospitals we studied. Percent positive scores are typically reported by culture researchers and show the percentage of respondents within a hospital who averaged a positive response (i.e. at least 4 out of 5 on the 5-point Likert-type scale) across all items in a scale. The percent positive results in Table 5 show that there is variability across the different types of culture we measured within this healthcare system.
Table 5.
Culture scores (percent positive) by hospital.
| Hospital | Number of units | Safety culture | Teamwork culture | Adherence to shared baselines | Preoccupation with failure |
|---|---|---|---|---|---|
| A | 24 | 73 | 75 | 65 | 80 |
| B | 29 | 78 | 77 | 76 | 84 |
| C | 8 | 65 | 71 | 63 | 68 |
| D | 42 | 75 | 75 | 77 | 85 |
| E | 33 | 76 | 77 | 80 | 81 |
| F | 29 | 78 | 78 | 81 | 89 |
| G | 16 | 88 | 88 | 79 | 90 |
| H | 27 | 83 | 81 | 78 | 91 |
| I | 42 | 77 | 77 | 77 | 85 |
| J | 17 | 86 | 84 | 79 | 86 |
| K | 23 | 78 | 75 | 80 | 85 |
| L | 9 | 79 | 73 | 70 | 80 |
| M | 78 | 71 | 73 | 68 | 80 |
Note: Mean scores reflect the average percent positive for the units within the specific hospital.
Discussion
Much has been discussed about HROs in other industries, and this study represents one of the first attempts of which we are aware to measure a specific HRO factor as postulated by Weick and Sutcliffe in a healthcare setting. We discovered that our survey items measured two cultural factors – preoccupation with failure and adherence to shared baselines – that had acceptable internal consistency and evidence for discriminant validity. Additionally, both cultural factors were signif-icantly correlated with overall patient safety grade (concurrent validity). Interestingly, preoccupation with failure was a stronger predictor of patient safety grade than safety culture and teamwork culture.
This study is important because clinicians and patient safety leaders can use our survey tool to begin assessing these new aspects of culture in their hospitals or work units. Similar to previous culture work we have completed,23 these factors can be used as stand-alone surveys or added to existing safety culture tools. Consistent with our work with the SAQ,23,24 we specifically designed the questions for this instrument so that it can be applied at the individual, the unit, and the hospital level. The high correlation with patient safety grade is of note. In combination with its ability to discriminate from other culture constructs, our new cultural factors add complementary and actionable information. For example, improvement in the shared baselines dimension might be accomplished by creating relevant monitoring systems based on item-level scores to the Adherence to Shared Baselines items. Preoccupation with failure might be improved by implementing a tool from the Comprehensive Unit-based Safety Program, called Learning from Defects, which tasks a dedicated caregiver with identifying, remedying, and providing staff feedback on potential patient safety hazards.25,26 Our survey provides practitioners with the opportunity to formally test such interventions. Given the lack of evidence, such efforts should be undertaken in ways that allow careful evaluation of their impact, and results should disseminated. Finally, the hospital-level results in Table 5 serve two purposes. From a research perspective, we can see that there is variability across the types of culture, which supports the notion that we are measuring different types of culture. Otherwise, we would expect all of the scores within a hospital to be the same across all types of culture. From a practical perspective, the percent positive results allow hospitals to know where they want to focus improvement efforts. For example, Hospitals A and I might focus on improving their Adherence score, while Hospitals B and D might focus on Preoccupation with Failure.
One limitation of our study is that all data came from one hospital system and we are not able to generalize our results beyond this system. Second, all data are based on provider perceptions, including the overall patient safety grade variable that we treated as an outcome. This, however, is congruent with current efforts at measuring unit or hospital culture and rooted in the field of aviation.15 Third, we do not know the demographics of those who did not respond. Fourth, we used listwise deletion as opposed to data imputation. Finally, we initially attempted to create items for each of these five dimensions and found it difficult to do so. It is possible that some aspects of HROs are better measured in ways other than via surveys.
Much work remains to be done in the area of high reliability as it pertains to patient safety. Based on the high means and low standard deviations of the first two preoccupation with failure items, these items need to be either revised or additional items need to be created to measure preoccupation with failure. Additionally, research needs to demonstrate that these cultural factors are correlated with clinical outcomes or other measures of hospital performance. Finally, future studies are needed to test new survey items that assess additional principles of high reliability culture, including sensitivity to operations, resilience, reluctance to simplify, and deference to expertise, to the extent that it is possible to create meaningful items for these dimensions.
The concept of HROs is becoming increasingly mentioned in the healthcare arena as a way for hospitals to continuously provide safe services.3,11 We created and tested one of the first surveys to measure one specific aspect and one complementary aspect of HROs in healthcare settings and found initial evidence for the reliability and validity of our survey. Our results provide an initial but promising step on the research path to understanding high reliability culture in healthcare.
Acknowledgments
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship and/or publication of this article: At the time this work was done, Dr. Thomas’s contribution was supported in part by Eunice Kennedy Shriver National Institute of Child Health and Human Development 1K24HD053771 (PI: Thomas) and Dr. Profit’s contribution was supported in part by the Eunice Kennedy Shriver National Institute of Child Health and Human Development #1 K23 HD056298 (PI: Profit).
Footnotes
Declaration of conflicting interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
References
- 1.Institute of Medicine. To err is human. Washington, DC: National Academy Press, 1999. [Google Scholar]
- 2.Wachter RM. Patient safety at ten: unmistakable progress, troubling gaps. Health Aff (Millwood) 2010; 29: 165–173. [DOI] [PubMed] [Google Scholar]
- 3.Chassin MR and Loeb JM. The ongoing quality improvement journey: next stop, high reliability. Health Aff (Millwood) 2011; 30: 559–568. [DOI] [PubMed] [Google Scholar]
- 4.Christianson MK, Sutcliffe KM, Miller MA, et al. Becoming a high reliability organization. Crit Care 2011; 15: 314. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Weick KE and Sutcliffe KM. Managing the unexpected: Assuring high performance in an age of complexity. San Francisco, CA: Jossey-Bass, 2001. [Google Scholar]
- 6.Sammer CE, Lykens K, Singh KP, et al. What is patient safety culture? A review of the literature. J Nurs Scholarship 2010; 42: 156–165. [DOI] [PubMed] [Google Scholar]
- 7.Clarke JR, Lerner JC and Marella W. The role for leaders of health care organizations in patient safety. Am J Med Qual 2007; 22: 311–318. [DOI] [PubMed] [Google Scholar]
- 8.Ketring SP and White JP. Developing a systemwide approach to patient safety: the first year. Jt Comm J Qual Improv 2002; 28: 287–295. [DOI] [PubMed] [Google Scholar]
- 9.Pronovost PJ, Weast B, Holzmueller CG, et al. Evaluation of the culture of safety: survey of clinicians and managers in an academic medical center. Qual Saf Health Care 2003; 12: 405–410. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Vogus TJ and Sutcliffe KM. The safety organizing scale: development and validation of a behavioral measure of safety culture in hospital nursing units. Med Care 2007;45: 46–54. [DOI] [PubMed] [Google Scholar]
- 11.Agency for Healthcare Research and Quality. Becoming a high reliability organization: operational advice for hospital leaders. AHRQ Publication No. 08–0022 Rockville, MD: Author, 2008. [DOI] [PubMed] [Google Scholar]
- 12.James BC and Savitz LA. How Intermountain trimmed health care costs through robust quality improvement efforts. Health Aff 2011; 30: 1185–1191. [DOI] [PubMed] [Google Scholar]
- 13.James BC and Lazar JS. Practice-based learning and improvement: a clinical improvement action guide. 2nd ed. Illinois, USA: Joint Commission Resources. [Google Scholar]
- 14.Swensen SJ, Meyer GS, Nelson EC, et al. Cottage industry to postindustrial care – the revolution in health care delivery. N Engl J Med 2010; 362: e12. [DOI] [PubMed] [Google Scholar]
- 15.Sexton JB, Helmreich RL, Neilands TB, et al. The Safety attitudes questionnaire: psychometric properties, benchmarking data, and emerging research. BMC Health Serv Res 2006; 6: 44. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Agency for Healthcare Research and Quality. Hospital survey on patient safety culture. Rockville, MD: Author, 2012. [Google Scholar]
- 17.Reise SP, Widaman KF and Pugh RH. Confirmatory factor analysis and item response theory: two approaches for exploring measurement invariance. Psychol Bull 1993;114: 552e66. [DOI] [PubMed] [Google Scholar]
- 18.Bentler PM and Bonnet DG. Significance tests and goodness of fit in the analysis of covariance structures. Psychol Bull 1980; 88: 588e–606. [Google Scholar]
- 19.Bentler PM. Comparative fit indexes in structural models. Psychol Bull 1990; 107: 238e46. [DOI] [PubMed] [Google Scholar]
- 20.Steiger JH and Lind J. Statistically based tests for the number of common factors In: The annual meeting of the psychometric society, Iowa City, IA, 30 May 1980. [Google Scholar]
- 21.Bollen KA. Structural equations with latent variables. New York: Wiley, 1989. [Google Scholar]
- 22.Kline RB. Principles and practice of structural equation modeling. New York: The Guilford Press, 2005. [Google Scholar]
- 23.Brown TA. Confirmatory factor analysis for applied research. New York: The Guilford Press, 2012. [Google Scholar]
- 24.Etchegaray JM and Thomas EJ. Comparing two safety culture surveys: safety attitudes questionnaire and hospital survey on patient safety. BMJ Qual Saf 2012; 21: 490–498. [DOI] [PubMed] [Google Scholar]
- 25.Berenholtz SM, Hartsell TL and Pronovost PJ. Learning from defects to enhance morbidity and mortality conferences. Am J Med Qual 2009; 24: 192–195. [DOI] [PubMed] [Google Scholar]
- 26.Pronovost PJ, Holzmueller CG, Martinez E, et al. A practical tool to learn from defects in patient care. Jt Comm J Qual Patient Saf 2006; 32: 102–108. [DOI] [PubMed] [Google Scholar]