Skip to main content
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2021 Sep 1.
Published in final edited form as: J Patient Saf. 2020 Sep;16(3):223–231. doi: 10.1097/PTS.0000000000000279

Patient Safety Culture Survey in Pediatric Complex Care Settings: A Factor Analysis

Amanda Hessels 1, Meghan Murray 2, Bevin Cohen 3, Elaine Larson 4
PMCID: PMC5648643  NIHMSID: NIHMS773558  PMID: 28426522

Introduction

Patient safety is a vital component of health care quality. Patient safety culture (PSC) of a healthcare organization is defined as “the product of individual and group values, attitudes, perceptions, competencies, and patterns of behavior that determine the commitment to, and the style and proficiency of, an organization’s health and safety management.”1 In hospitals and adult outpatient settings a culture of safety, characterized by shared perspectives of the importance of safety, fidelity and transparency of communications, and shared confidence in the efficacy of preventive measures, has been associated with improved patient outcomes.14 Standardized, valid and reliable tools exist to measure PSC in hospitalized adults and children, and adults in non-hospital settings, such as the Safety Attitudes Questionnaire, Safety Organizing Scale, Survey on Resident Safety in Nursing Homes, Hospital Survey on Patient Safety Culture and Nursing Home Survey on Patient Safety Culture (NHSPSC), but no tool has been tested for its relevance to pediatric long term care (pLTC) settings which are unique and have distinctly different cultures from other types of health care delivery systems.58

In the past decade there has been a steady increase in the number and complexity of children with complex medical conditions, some of whom reside in an estimated 100 pLTC facilities in the U.S.9,10 Estimates of the number of children with special health care needs in the U.S. are as high as 11 million.11 Estimates of those residing in pLTC with complex medical conditions are variable dependent upon definitions of condition, care setting or data repository used, but include some of the estimated 29,000 children in congregate care facilities (e.g. skilled nursing, long term care, or residential care), such as an estimated 4,886 children with special health care needs residing in skilled nursing facilities.12,13 These facilities describe a setting where the children live, and as such children are referred to as “residents”. This is in contrast to settings such as transitional rehabilitation, residential school and other settings that are time-limited and do not provide higher level care services.12 Children with complex medical conditions have intense care needs, typically require 24-hour skilled nursing procedures and are dependent on technology for activities of daily living and thus reside in pLTC.12,13

These children are considerably functionally and or developmentally impaired and may have medical diagnoses such as congenital anomalies, cystic fibrosis, multi-system disease, oncologic disease or other conditions. Care needs depend on multiple factors, including age, developmental stage, need for mechanical ventilation, orthotic assistance, and other invasive therapies requiring technological and functional support and services to manage life-long chronic clinical conditions and sequelae.12, 13 Consequently, these children require intense care, utilize a sizeable amount of resources, and often transition between pLTC and hospital settings to receive care for exacerbations of conditions.9, 1113

Healthcare workers in pLTC settings function in a hybrid acute care-residential setting and have unique workflow patterns, priorities and relationships among themselves and with residents and their family members.9, 10 Many different types of staff work these settings such as nurses, physicians, and nursing aides; respiratory, recreational, physical and occupational therapists; teachers and teachers’ aides; and environmental service workers and social workers. In these home-like settings there is a considerable interaction among children and direct and close contact between children, staff, families, volunteers, and visitors.9, 10 Similar to adult LTC facilities, hospital level care is provided to residents who require 24-hour skilled nursing care for long periods of time.12

As such, understanding the nature and extent that factors influence healthcare workers’ behaviors in pLTC settings is important for patient outcomes and may influence healthcare utilization, such as re-hospitalization rates. PSC is one such factor has been identified as essential in all settings that render care to pediatrics to eliminate avoidable harm, such as adverse drug events, health care associated infections and errors of care.14 Patient safety problems and solutions are described as unique for children as developmental, physical and legal status issues are unique and multifactorial, though notably studies on pediatric safety are limited in non-hospital settings.14

Synthesizing literature regarding the importance of PSC in adult populations in nursing home and critical care settings and given the emerging issues of an increased number of children with complex medical conditions in congregate care settings (such as pLTC) and policy statements from the American Academy of Pediatrics regarding patient safety, we believe PSC may be an important construct in pLTC and merit measurement. Typical safety events in pLTC, adverse events and critical incidents are not currently subject to standardized reporting mechanisms and measures, such as quality indicators or patient safety indicators in hospitals, thus limiting prevalence estimations and benchmarking. We believe a measure of PSC may serve as an intervention to raise staff awareness of safety issues, as a means to evaluate safety improvement initiatives, and as a diagnostic tool of safety culture as facilities track and trend case and event reporting rates within their facilities. That is, it is likely the PSC is important in these settings as well, and identifying the role of the PSC in pLTC could guide organizational improvement.

Yet it is unknown if existing tools adequately capture the PSC construct in these settings. As these pediatric setting and provider characteristics that influence team functioning are similar to adult LTC, the NHSPSC was selected as opposed to the hospital survey. This tool was selected following face and content validity screening to evaluate the dimension and item relevancy by two pediatric physicians, who are both clinicians and researchers, familiar with the setting. The tool was also familiar to the leadership of the facilities. The purpose of this study was to evaluate the construct validity through factor analysis of the Agency for Healthcare Research and Quality (AHRQ) NHSPSC tool slightly modified for use in the pLTC setting.

Methods

Sample and Setting

This study was part of a larger parent project funded by the Agency for Healthcare Research and Quality (AHRQ, Keep it Clean for Kids (KICK), R01HS021470). This four year (2012–2016) research project was designed to improve infection prevention practices and PSC in three metropolitan New York area pLTC facilities with 54, 97 and 137 beds. Residents have a variety of complex and chronic medical conditions: 77–85% have a feeding tube (e.g., gastrostomy, nasogastric tube, jejunal tube), 29–51% have a tracheostomy, and 60–74% are non-ambulatory. Each site provides a wide range of medical, educational (e.g., on-site Department of Education school), and therapeutic services including respiratory, physical, occupational, recreational, music, and art therapies. The number of staff at each site ranges from 244 to 539. Inclusion criteria of the convenience sample included all staff available (e.g. nurses, administrative, school, therapists, housekeeping personnel) and willing to participate in this pen and paper survey, no site personnel were excluded. The study was approved by the Columbia University Medical Center Institutional Review Board (IRB) and the relevant ethics review group at each facility.

Original Instrument

The NHSPSC is a 44-item survey designed for staff and administrators of nursing homes facilities; it was not designed for use in assisted living, community care or independent living facililities.2,15 Responses are measured on a 5-point Likert scale ranging from strongly disagree to strongly agree, or never to always for frequency reporting. Briefly, tool development included a review of the literature on resident safety in nursing homes, health care quality, medical errors, error reporting, safety climate and culture, and organizational climate and culture. In addition, a review of existing nursing home surveys was conducted and none were identified focused specifically on staff opinions about resident safety issues. Researchers consulted with more than two dozen experts in the field of nursing home safety to identify key topics and issues. They identified a potential list of dimensions to include in the survey and sought consensus from long-term care experts. A draft of survey items was developed to measure the key dimensions and iteratively pretested with nursing home staff to ensure that the items were easy to understand and to answer and were relevant to resident safety in nursing homes. The pilot test version of the NHSPSC was completed in 2007 by 3,700 staff working in 40 nursing homes across the U.S. Participating nursing homes varied by bed size, geographic region, urban or rural setting, and ownership.2

Following individual item analysis the tool was again revised to ensure that the survey has sound psychometric properties; upon further examination 42 items were grouped into 12 conceptually distinct dimensions. The 12 dimensions include teamwork, staffing, compliance with procedures, training and skills, non-punitive response to mistakes, handoffs, feedback and communication about incidents, communication openness, supervisor expectations and actions promoting resident safety, overall perceptions of resident safety, management support of resident safety and organizational learning. The remaining two items asked respondents to assign an overall patient safety grade and number of events reported in the past 12 months.2

Administration of the Instrument

Prior to administration of the tool two pediatric physicians who practice in both hospital and pLTC settings served as content experts and confirmed the relevancy of the NHSPSC dimensions and items for the pLTC setting. In this study, while the items are relevant to a pediatric setting, wording of this instrument was modified for the pediatric setting (e.g., the term ‘this nursing home’ was replaced by ‘this facility’). The modified survey used in this study is named the NHSPSC-pLTC (Table 1) and was administered following the AHRQ Survey User’s Guide on Patient Safety Culture.15 In fall, 2012 the paper-based surveys were distributed on each unit at each facility for one day shift and one night shift to all staff, both direct and indirect care staff, including on-site school personnel. Staff self-identified in the following categories: (1) Administrator/Manager (e.g. Executive Director/Administrator, Medical Director, Director of Nursing/Nursing Supervisor, Department Head, Unit Manager/Charge Nurse, Assistant Director/Assistant Manager, Minimum Data Set (MDS) Coordinator/Resident Nurse Assessment Coordinator (RNAC)); (2) Physician (MD, DO); (3) Licensed Nurse (Nurse Practitioner (NP), Clinical Nurse Specialist (CNS), Registered Nurse (RN), Licensed Practical Nurse (LPN), Wound Care Nurse); (4) Nursing Assistant/Aide (Certified Nursing Assistant (CNA), Nursing Aide/Nursing Assistant); (5) Other Care Staff (Activities Staff Member, Dietitian/Nutritionist, Medication Technician, Pastoral Care/Chaplain, Pharmacist, Podiatrist, Social Worker, Physical/Occupational/Speech/Respiratory Therapist); (6) Administrative Support Staff (Administrative Assistant, Admissions, Billing/Insurance, Secretary, Human Resources, Medical Records) or (7) Support Staff (Drivers, Food Service/Dietary, Housekeeping, Laundry Service, Maintenance, Security).

Table 1.

Dimensions, items and reliability of the NHSPSC-pLTC used in the present study

1. Teamwork: published reliability of this dimension—Cronbach’s alpha (4 items) = .86, this sample = .84
 A1. Staff in this facility treat each other with respect.
 A2. Staff support one another in this facility.
 A5. Staff feel like they are part of a team.
 A9. When someone gets really busy in this facility, other staff help out.
2. Staffing: published reliability of this dimension—Cronbach’s alpha (4 items) = .71, this sample = .55

 A3. We have enough staff to handle the workload.
 A8. Staff have to hurry because they have too much work to do (negatively worded).
 A16. Children’s needs are met during shift changes
 A17. Because so many staff quit their jobs, there are few experienced staff to provide safe care (negatively worded).
3. Compliance With Procedures: published reliability of this dimension—Cronbach’s alpha (3 items) = .73, this sample = .68

 A4. Staff follow standard procedures to care for children.
 A6. Staff use shortcuts to get their work done faster (negatively worded).
 A14. To make work easier, staff often ignore procedures (negatively worded).
4. Training & Skills: published reliability of this dimension—Cronbach’s alpha (3 items) = .76, this sample = .72

 A7. Staff get the training they need in this facility.
 A11. Staff have enough training on how to handle difficult children.
 A13. Staff understand the training they get in this facility.
5. Nonpunitive Response to Mistakes: published reliability of this dimension—Cronbach’s alpha (4 items) = .74, this sample = .74

 A10. Staff are blamed when a child is harmed (negatively worded).
 A12. Staff are afraid to report their mistakes (negatively worded).
 A15. Staff are treated fairly when they make mistakes.
 A18. Staff feel safe reporting their mistakes.
6. Handoffs: published reliability of this dimension—Cronbach’s alpha (4 items) = .86, this sample = .83

 B1. Staff are told what they need to know before taking care of a child for the first time.
 B2. Staff are told right away when there is a change in a child’s care plan.
 B3. We have all the information we need when children are transferred from the hospital.
 B10. Staff are given all the information they need to care for children.
7. Feedback & Communication About Incidents: published reliability of this dimension—Cronbach’s alpha (4 items) = .85, this sample = .84

 B4. When staff report something that could harm a child someone takes care of it.
 B5. In this facility, we talk about ways to keep incidents from happening again.
 B6. Staff tell someone if they see something that might harm a child.
 B8. In this facility we discuss ways to keep children safe from harm.
8. Communication Openness: published reliability of this dimension—Cronbach’s alpha (3 items) = .84, this sample = .85

 B7. Staff ideas and suggestions are valued in this facility.
 B9. Staff opinions are ignored in this facility (negatively worded).
 B11. It is easy for staff to speak up about problems in this facility.
9. Supervisor Expectations & Actions Promoting Resident Safety: published reliability of this dimension—Cronbach’s alpha (3 items) = .81, this sample = .82

 C1. My supervisor listens to staff ideas and suggestions about the safety of children.
 C2. My supervisor says a good word to staff who follow the right procedures.
 C3. My supervisor pays attention to child safety problems in this facility.
10. Overall Perceptions of Resident Safety: published reliability of this dimension—Cronbach’s alpha (3 items) = .86, this sample = .84

 D1. Children are well cared for in this facility.
 D6. This facility does a good job keeping children safe.
 D8. This facility is a safe place for children.
11. Management Support for Resident Safety: published reliability of this dimension—Cronbach’s alpha (3 items) = .83, this sample = .84

 D2. Management asks staff how the facility can improve safety for children.
 D7. Management listens to staff ideas and suggestions to improve safety for children.
 D9. Management often walks around the facility to check on care for children.
12. Organizational Learning: published reliability of this dimension—Cronbach’s alpha (4 items) = .81, this sample = .72

 D3. This facility lets the same mistakes happen again and again (negatively worded).
 D4. It is easy to make changes to improve safety for children in this facility.
 D5. This facility is always doing things to improve safety for children.
 D10. When this facility makes changes to improve safety for children, it checks to see if the changes have worked.
13. Overall Ratings

 E1. I would tell friends that this is a safe facility for their children.
 E2. Please give this facility an overall rating on safety for children.

Note: “NHSPSC” is Nursing Home Survey on Patient Safety Culture; “p-LTC” is pediatric long term care facility

Surveys were distributed at the beginning of the shift and collected at the end of the shift, or staff had the option of retuning them to the mailbox of the staff member responsible for infection control. All staff were instructed that the survey was voluntary and anonymous. Staff who returned a survey were eligible for inclusion in a raffle for several $5 coffee shop gift cards per site. The list for the raffle was completely separate from the surveys.

Statistical Analysis

Eight of the 42 items that comprised the 12 dimensions were negatively worded and reverse coded for analysis. All items were examined in the factor analyses performed. Statistical analysis was performed using STATA/MP 12.1 software (StataCorp LP, College Station, TX). Since the number of subjects who completed the survey was greater than 200, and approximated five times the total number of items used for factor analysis (N:p ratio 5:1), missing data was not imputed in analysis.16 Complete data on all items was available for 125 respondents demonstrating a sufficient N: p ratio of 3:1 for the set of items. The overarching approach was to test the a priori model of the original structure using confirmatory factor analysis (CFA).and then if the model does not adequately fit the data examine the current structure through exploratory factor analysis (EFA)16 The study follows comprehensive reporting guidelines and recommendations.17

Sampling adequacy was further assessed using Kaiser-Meyer-Olkin (KMO) and anti-image correlation matrix procedures. Items that comprised each construct were then examined for the a priori identified 12 dimensions using Cronbach’s alpha for internal consistency. A correlation matrix of all items in the scale was generated and an initial un-rotated factor solution was extracted. Finally, principal factor analysis was used, employing both oblique (Promax) and orthogonol (Varimax) rotation, to maximize the independence of the factors and enhance theoretical clarity of the underlying survey structure.

Communality to identify the variance shared by one or more variables in the factors and uniqueness was ascertained to measure the variance in single variables uncorrelated with the component factors.16, 18, 19 Only factors with an Eigenvalue greater than or equal to 1.0 were retained in all model construction and a strict factor loading cutoff of 0.40 was used. The dimensions and items, published Cronbach’s alpha reliability coefficients and those calculated from this study sample of the NHSPSC-pLTC are shown in Table 1.

Results

A total of 239 completed surveys, were pooled from three settings for use in this analysis. It was not possible to record the number of surveys distributed to this convenience sample within the facilities as surveys were distributed in person and site liaisons also made copies and redistributed in hard copy and via email; therefore we cannot estimate the response rate during our sampling period. However, we do estimate there are approximately 1100 total staff employed at the facilities.

The majority of respondents from whom data were available worked directly with children “most of the time” (83%), primarily in roles of licensed nurses (42%), nursing assistants/aides (22%) or other care staff (activities staff, dietitian/nutritionist, medication technician, pastoral care/chaplain, pharmacist, physical/occupational/speech/respiratory therapist, podiatrist, social worker). Data was also available from those who responded in administrator/management, physician, administrative care staff and support staff categories. There were no categories that were not represented. Respondents reported being employed in their respective facilities for 2–11 months (26%), or 6–10 years (19%), 11 years or more (21%), where they worked 25–40 hours per week (67%) or more (24%). The majority (57%) of respondents work day, followed by night (29%) and evening (14%) shifts.

Though the total number of missing items was low, and the number missing per item was also low, missing responses were evident. Missing values at the item level due to blank or invalid responses and “doesn’t apply or don’t know” ranged from one missing (0.004%) to 25 missing (10.4%). Missing data from these respondents was examined for patterns and by respondent type and item to allow for additional appraisal of lines of evidence for the relevant subgroups in the intended population for this instrument.20 Those who selected “don’t know/doesn’t apply” to at least one item included approximately 21% of nurses, 36% of nursing aides and 45% of the “other staff,” or by tenure at facility approximately 46% of those employed 2–11 months, 19% of those employed 6–10 years, and 15% of those employed 11 years or longer. At least one missing item was found in surveys completed by approximately 40% of nursing assistants/aides, 23% of nurses, and 18% of “other staff;” or by tenure at facility at least one missing item in 28% of those employed 2–11 months, 25% of those employed 6–10 years, and 42% of those employed 11 years or longer. All items had missing data, therefore the sample size to test each dimension, and subsequently the overall construct, was impacted by missing responses.

Matrix Analysis

Correlation matrices were generated to identify relationships among the items in this study. Bartlett’s test of sphericity (χ2 = 3468.1; df = 861; p<0.001) indicated sufficiency of inter-item correlations, the KMO measure of sampling adequacy was 0.85 and deemed “meritorious” on the recognized KMO rating, and the anti-image correlation matrix values were closer to zero.18,19,21,22 Based upon these three tests, and in consideration of a priori knowledge of the underlying structure of the construct, we considered that the data and matrix were appropriate for factor extraction and all items were included in the analysis.

Confirmatory Analysis

The CFA was performed using maximum likelihood structural equation modeling techniques without constraints on the 42 items in the a priori identified 12 dimensions. Goodness-of-fit indices were then calculated including the chi-square index of discrepancy between the model fit and data, commonly known as an “exact fit index,” and indicated a poor fit by χ 2 likelihood ratios (χ 2 = 1575.57; df = 594; p <0.001). The Root Mean Square Error of Approximation (RMSEA) takes both sample size and error approximation into account and demonstrated a poor fit with the data, (RMSEA = 0.119), (cutoff is less than 0.06), as did the comparative fit index (CFI = 0.54), the non-normed Tucker-Lewis index (TLI = 0.51), (cutoff is >-0.90 for CFI and TLI) and the standardized root mean squared residual (SRMR = 0.29) (perfect fit is 0, good fit is less than 0.10) 22,23 The coefficient of determination (CD) index, considered an R2 for the model, indicated a good fit (CD = 1) where a value close to one indicates a good fit. 24

This model identified a poor correlation between the observed variables and factors as follows. A4 (“staff follow standard procedures to care for children”) loaded just above threshold with compliance with procedures (0.42), (R2 = 0.18), A16 (“children’s needs are met during shift changes”) loaded poorly on staffing (0.3) (R2 = 0.09), as did D3 (“this facility lets the same mistakes happen again and again”) on organizational learning (0.34) (R2 = 0.12). The low R2 values indicate the items do not fit well with the model. However, the factor loading matrix item A12 (“staff are afraid to report their mistake”) performed well with a factor loading of 0.77 (R2 = 0.59), as did A8 (“staff have to hurry because they have too much work to do”) with a factor loading of 0.67 (R2 = 0.45). Finally, modification indices were conducted and suggest that several variables manifest the same latent variables including A11 and B1, A10 and A15, A12 and A18, and A9 and D10.

In consideration of both the difference between the number of respondents who completed all items in all dimensions (125) and all items per dimension (range 190 – 227) and our aim to test the structure of each factor, we then treated each dimension independently as a one-factor model. This allowed us to better discriminate the relationship among the observed variables that comprised a distinct a priori underlying construct. Goodness-of-fit tests were conducted, and as shown in Table 2, eight of the 12 one-factor models were a good fit as indicated by four or more goodness-of-fit statistics, demonstrating construct validity of these factors. The models that showed a poor fit included staffing, non-punitive response to mistakes, communication openness and organizational learning.

Table 2.

Summary of NHSPSC 12 factor goodness-of-fit statistics

Factor (# items, # respondents) Model vs. saturated Chi-Square (df) P value RMSEA CFI TFI CD
Teamwork (4, 223) 4.36(2) 0.11* .073 0.99* 0.98* 0.90*
Staffing (4, 201) 8.42(2) 0.02 0.13 0.94* 0.82 0.68
Comply with Procedures (3, 211) 0.00(0) 0.00 0.00* 1.00* 1.00* 0.77*
Training and Skills (3, 210) 0.00(0) 0.00 0.00* 1.00* 1.00* 0.73*
Non-punitive response to mistakes (4, 190) 42.13(2) 0.00 0.33 0.78 0.35 0.76*
Handoffs (4, 216) 1.71(2) 0.43* 0.00* 1.00* 1.00* 0.84*
Feedback and incident communication (4, 215) 2.39(2) 0.30* 0.03* 0.99* 1.00* 0.85*
Communication openness (3, 212) 0.00(0) 0.00 1.00 1.00* 0.86* 0.82*
Supervisor expectations and actions (3, 221) 0.00(0) 0.00 0.00* 1.00* 1.00* 0.86*
Overall perceptions of safety (3, 227) 0.00(0) 0.00 0.00* 1.00* 1.00* 0.90*
Management support (3, 204) 0.00(0) 0.00 0.00* 1.00* 1.00* 0.86*
Organizational learning (4, 209) 13.31(2) 0.00 0.16 0.94* 0.81 0.77*
*

Acceptable model indices: Chi-Square value near zero and p >.05, Root Mean Square Error of Approximation (RMSEA) <0.06, and Comparative Fit Index (CFI) and Tucker-Lewis Fit Index (TFI) ≥0.90 and Coefficient of determination (CD) ≥.70 (where 1 indicates a perfect fit)

Factor Extraction

The EFA results confirmed the existence of multiple factors or dimensions, providing evidence that suggested many of the a priori item groupings did fall into distinct factors, affirming the dimensionality of the construct. Based upon the testing of this instrument in adult nursing home settings we expected a 12 factor solution. An un-rotated principal factors analysis was conducted on the 42 items identifying a total of eight factors with Eigenvalues greater than or equal to 1.

The factor loadings, or pattern matrix, were calculated and the majority of variables sufficiently demonstrated a good fit by unique variances communality >0.40, with exception of items A3 (0.31), A8 (0.39), A16 (0.25), A4 (0.36), and D3 (0.33), indicating these variables share more variance with other variables in the factor model. A scree plot was then constructed which indicated a break in Eigenvalues at factor number four, indicating a three factor solution, and again at factor number 9, indicating an 8 factor solution. Based upon the aims of this study we proceeded with analysis using the eight factor solution.

Factor Rotation

Two types of rotation were employed to better interpret the eight factor structure: the orthogonal Varimax and the oblique Promax rotations. Factors loaded identically in both methods indicating the orthogonal method was well suited and chosen for the remaining analysis. The rotated component matrix and factors showing items with loading factors above 0.4 is shown in Table 3. Four items did not have a loading factor above 0.40 (A4, D3, A8 and A16), and two items (A12 and B7) loaded on two factors.

Table 3.

Extracted variable factor loadings and matrix (n= 125)

Item Factors and factor loadings
1 2 3 4 5 6 7 8
A1. Staffs in this facility treat each other with respect. 0.68
A2. Staff support one another in this facility. 0.83
A5. Staff feel like they are part of a team. 0.66
A9. When someone gets really busy in this facility, other staff help out. 0.57
A3. We have enough staff to handle the workload. 0.41
A17. Because so many staff quit their jobs, there are few experienced staff to provide safe care. 0.46
A6. Staff use shortcuts to get their work done faster. 0.56
A14. To make work easier, staff often ignore procedures. 0.61
A7. Staff get the training they need in this facility. 0.65
A11. Staff have enough training on how to handle difficult residents. 0.65
A13. Staff understand the training they get in this facility. 0.57
A10. Staff are blamed when a child is harmed. 0.67
A12. Staff are afraid to report their mistakes. 0.56 0.47
A15. Staff are treated fairly when they make mistakes. 0.55
A18. Staff feel safe reporting their mistakes. 0.63
B1. Staff are told what they need to know before taking care of a child for the first time. 0.54
B2. Staff are told right away when there is a change in a child’s care plan. 0.56
B3. We have all the information we need when residents children are transferred from the hospital. 0.44
B10. Staff are given all the information they need to care for children. 0.59
B4. When staff report something that could harm a child someone takes care of it. 0.58
B5. In this facility, we talk about ways to keep incidents from happening again. 0.78
B6. Staff tell someone if they see something that might harm a child. 0.69
B8. In this facility, we discuss ways to keep children safe from harm. 0.68
B7. Staff ideas and suggestions are valued in this facility. 0.61 0.45
B9. Staff opinions are ignored in this facility. 0.63
B11. It is easy for staff to speak up about problems in this facility. 0.67
C1. My supervisor listens to staff ideas and suggestions about the safety of children 0.77
C2. My supervisor says a good word to staff who follow the right procedures. 0.61
C3. My supervisor pays attention to child safety problems in this facility. 0.77
D1. Residents are well cared for in this facility. 0.64
D6. This facility does a good job keeping children safe. 0.75
D8. This facility is a safe place for children. 0.79
D2. Management asks staff how the facility can improve the safety for children 0.53
D7. Management listens to staff ideas and suggestions to improve the safety for children. 0.54
D9. Management often walks around the facility to check on care for children. 0.73
D4. It is easy to make changes to improve resident safety in this facility. 0.55
D5. This facility is always doing things to improve the safety for children. 0.73
D10. When this facility makes changes to improve safety for children, it checks to see if the changes have worked. 0.69

Factor reliability and independence

Cronbach’s alpha coefficients were used to measure internal consistency of the group of items measuring the a priori identified 12 dimensions. Good to very good reliability by alpha coefficient was found in all dimensions, ranging from 0.72 to 0.85, with exception of the dimension compliance with procedures deemed acceptable (α = 0.67) and staffing deemed poor (α = 0.55).25 This was also conducted with the identified eight factor solution and findings were either consistent (compliance with procedures deemed acceptable,α = 0.68), or demonstrated improved internal consistency with Cronbach’s alpha coefficients ranging from 0.79 to 0.88. The internal consistency was also examined by average inter-item correlations to determine the effect of removal or addition of items from the NHSPSC-pLTC to each of the eight identified factors as shown in Table 4. High inter-item correlations were demonstrated, all sufficiently above acceptable cut-off value of 0.30, indicating the items are related to the same construct.18, 25 Affirming the factors are distinct and uncorrelated, the clean pattern matrix was clean; only two of the 42 items cross-loaded on more than one factor. Finally, the standardized loadings that represent the correlation between the observed variable and the latent factor were examined and R2 was calculated for each of the observed variables to identify the amount of variance accounted for by the latent variables.17, 18,26 Of all the items in the extracted eight factor model, the least item variance accounted for was D3 (“this facility lets the same mistakes happen again and again”) by R2 = 0.22, indicating this item did not fit well with the factor organizational learning and perhaps taps a dimension of organizational learning not captured by the other variables.

Table 4.

Differences in composition of AHRQ NHSPSC and NHSPSC-pLTC

AHRQ NHSPSC Factor Structure NHSPSC-pLTC Factor Structure
Factor (#items) Items Factor reliability (α) Average inter-item correlation Factor Items Factor reliability (α) Average inter-item correlation
Teamwork (4) A1, A2, A5, A9 0.84 0.57 Teamwork and staffing (5) A1, A2, A5, A9, A3 0.82 0.48
Staffing (4) A3, A8, A16, A17 0.55 0.24
Compliance with Procedures (3) A4, A6, A14 0.68 0.41 Compliance with procedures (2) A6, A14 0.68 0.52
Training and Skills (3) A7, A11, A13 0.72 0.47 Training, skills and handoffs A7, A11, A13, B1, B3 0.79 0.42
Non-punitive response to mistakes (4) A10, A12, A15, A18 0.74 0.42 Non-punitive response, communication openness and management support and staffing (9) A10, A12, A15, A18, B7, B9, B11, D2, A17 0.88 0.46
Handoffs (4) B1, B2, B3, B10 0.83 0.54
Feedback and incident communication (4) B4, B5, B6, B8 0.84 0.57 Feedback and communication and handoffs (6) B4, B5, B6, B8, B2, B10 0.86 0.50
Communication openness (3) B7, B9, B11 0.85 0.66
Supervisor expectations and actions (3) C1, C2, C3 0.82 0.61 Supervisor expectations and actions (3) C1, C2, C3 0.82 0.61
Overall perceptions of safety (3) D1, D6, D8 0.84 0.64 Overall perceptions and organizational learning (5) D1, D6, D8, D4, D5 0.86 0.56
Management support (3) D2, D7, D9 0.83 0.63 Management support and organizational learning (3) D7, D9, D10 0.80 0.58
Organizational learning (4) D3, D4, D5, D10 0.73 0.40

Note: “NHSPSC” is Nursing Home Survey on Patient Safety Culture; “p-LTC” is pediatric long term care facility.

*

Bold AHRQ NHSPSC factors indicate poor goodness- of- fit statistics; bold NHSPSC- pLTC items indicate where differences occur between factor structures. Items .A4, D3, A8 and A16 are not included in the NHSPSC-pLTC due to factor loadings <0.40, items A12 and B7 loaded on two factors and are placed in the factor with the highest loading.

Discussion

This study examined aspects of the psychometric properties of the NHSPSC modified for use in pLTC settings. The factors extracted from the data were generally consistent with those in the literature for the HSOPSC.27- our knowledge only one other study has examined the psychometric properties of the NHSPSC, using a Swiss version.30 Similar to the Zúñiga et al. (2013)30 study, our analysis suggests a reduction from 12 factors to eight, and though the items comprising the factors are very similar to the parent NHSPSC structure, items did shift. Only one factor was identical between the original NHSPSC and the NHSPSC-pLTC, supervisor expectations and actions. The greatest change by number of items loaded was the addition of five items from the dimensions communication openness and management support to the dimension of non-punitive response to mistakes. This means in the pLTC setting these are not distinctly different factors, as based on this statistical analyses, and as proposed in the NHSPSC based on theory driven expert opinion for the adult settings.2, 8 Conceptually, the items measure the bidirectional relationship of the perception of communication tone and support from management to frontline workers and the perceived value and engagement in communication from frontline workers to management. This suggests a relationship between staff engagement in safety that is closely intertwined with staff perceptions of being blamed or treated fairly when they make mistakes, events happen or they provide information to leadership.

Perhaps the most unexpected finding was the poor factor loading of two of the four items comprising staffing, which were thus eliminated from the extracted model. The remaining two items A3”We have enough staff to handle the workload” and A17 “Because so many staff quit their jobs, there are few experienced staff to provide safe care,” loaded with teamwork and non-punitive response to mistakes, respectively. This suggests that a unique construct of staffing adequacy as measured by these items is not evident in the pLTC setting, though components of perceptions of staffing are related to staff perceptions of teamwork and a non-punitive environment. This finding is consistent with that identified by Zúñiga et al., 30 who examined the factor structure in adult nursing homes. Further, staffing may be a multi-dimensional concept in pLTC settings as it relates to safety; notably the items that were omitted due to factor loadings ≤0.40 include A16 “Children’s needs are met during shift changes,” and A8 “Staff have to hurry because they have too much work to do.” It appears the items pertaining to the staffing adequacy by number of personnel are relevant, but those that ascertain the effect of workflow and conditions under which the number of staff operate are inadequate. However, this requires further exploration in other pLTC settings as these omitted items are conceptually relevant to the PSC construct.

The original NHSPSC factors identified and tested in adult settings did not fit exactly with the data from pLTC settings in this study, despite good internal reliability of the dimensions. This may suggest the underlying theoretical structure may not adequately capture the perceptions of the PSC by healthcare workers in pLTC settings. Following factor extraction and rotation four items (A4, D3, A8 and A16) were omitted from further statistical testing, resulting in a better fit of a more parsimonious model of eight factors. While these items do appear to cover core PSC issues and are theoretically congruent, our interpretation is based upon our stated approach using a strict factor loading cutoff threshold of 0.40. Reasons for the low factor loadings may be related to other items more strongly correlated to the latent variables that better exemplify the construct of interest in this sample, rather than the item being irrelevant to the construct of PSC. Therefore, our findings suggest that this tool in modified format, including the items excluded after the EFA and using the subscales in modified form, may be appropriate for this population and setting in future surveys. This supports the pursuit of additional lines of evidence to support the validity and reliability of the instrument to measure PSC in pLTC.20

There are limitations to this study including face validity testing restricted to physicians, a possible non-response bias and a relatively small sample size of respondents to all items in the full scale. Missing values at the item level due to blank or invalid responses and “doesn’t apply or don’t know” ranged from one missing (0.004%) to 25 missing (10.4%). Because of this loss of responses for items that comprise all factors, items that comprised each factor were examined consistent with factor analysis procedures to include only those that responded to all items in a factor and omitted those that did not. Thus, though each dimension may be individually confirmed, the full model in the EFA employed a limited sample size thus limiting our ability to explore the PSC construct in this sample, as the grouping of items are based on the EFA results. That is, each dimension was tested, but the full construct testing of PSC was limited by the sample size that completed all items. As shown in Table 2, each original NHSPSC factor included at least three to four items with 190 to 227 complete values for analysis, indicating model testing requirements were met.21 Therefore, any limitation in sample size is anticipated to be minimal as complete responses were available from the majority of respondents per factor (80–95%).

Finally, analytic precision may be compromised by pooling data from three sites if there is considerable variation in perception of PSC. In this survey nearly 60% of the data came from one site with an equal amount from the other two sites and there was a sizeable proportion of respondents who had worked for either a very limited or very lengthy time in their respective facilities. In this factor analysis, however, the heterogeneity may be viewed as strength.

Conclusion

This study demonstrates that the AHRQ NHSPSC may be appropriate for pLTC settings. This is the first psychometric testing of this instrument in this population and warrants more robust testing to conclude if it is a sound instrument to measure PSC in this population. In this study we conducted a CFA to assess whether the adapted instrument had a similar structure to the original instrument and then an EFA as the structure could not be confirmed for several subscales. Further testing should include additional construct validity testing with sufficient response rates in a larger dataset to test the proposed model structure found in the EFA. Following any revisions, the next step would be to conduct a larger sample CFA and determine if the results are reliable across settings and populations. Additionally, further testing is needed to establish more broadly its’ content validity by examining if the scale includes items relevant for all professional groups in pLTC. Discriminant validity testing should also be conducted to assess if the scale actually measure something different than other instruments that measure simply teamwork or leadership. Finally, the scale must be examined in relationship to safety outcomes in pLTC to establish the relevance and usefulness of the tool in practice. Ultimately, increased use of this tool may yield data for benchmarking purposes across these specialized settings to inform frontline workers and organizational leaders of areas of strength and opportunity for improvement.

Acknowledgments

Source of Funding: Amanda Hessels was supported as a post-doctoral trainee by the National Institute of Nursing Research, National Institutes of Health (Training in Interdisciplinary Research to Prevent Infections), T32 NR013454. This study was part of a larger parent project funded by the Agency for Healthcare Research and Quality (Keep it Clean for Kids (KICK), R01HS021470).

Footnotes

Conflicts of Interest and Source of Funding:

Amanda Hessels was supported as a post-doctoral trainee by the National Institute of Nursing Research, National Institutes of Health (Training in Interdisciplinary Research to Prevent Infections), T32 NR013454. This study was part of a larger parent project funded by the Agency for Healthcare Research and Quality (Keep it Clean for Kids (KICK), R01HS021470).

Contributor Information

Amanda Hessels, Columbia University, School of Nursing.

Meghan Murray, Columbia University, School of Nursing.

Bevin Cohen, Columbia University, School of Nursing.

Elaine Larson, Columbia University, School of Nursing.

References

  • 1.Sorra JS, Nieva VF. Hospital Survey on Patient Safety Culture. Agency for Healthcare Research and Quality; Rockville, MD: 2004. Sep, AHRQ Publication No. 04-0041. http://www.ahrq.gov/qual/patientsafetyculture/ [Google Scholar]
  • 2.Sorra J, Franklin M, Streagle S. Nursing Home Survey on Patient Safety Culture. Agency for Healthcare Research and Quality; Rockville, MD: 2008. [Google Scholar]
  • 3.DiCuccio MH. The relationship between patient safety culture and patient outcomes: A systematic review. J Patient Saf. 2015;11(3):135–142. doi: 10.1097/PTS.0000000000000058. [DOI] [PubMed] [Google Scholar]
  • 4.Mardon RE, Khanna K, Sorra J, et al. Exploring relationships between hospital patient safety culture and adverse events. J Patient Saf. 2010;6(4):226–32. doi: 10.1097/PTS.0b013e3181fd1a00. [DOI] [PubMed] [Google Scholar]
  • 5.Sexton JB, Helmreich RL, Neilands TB, Rowan K, Vella K, Boyden J, … Thomas EJ. The safety attitudes questionnaire: Psychometric properties, benchmarking data, and emerging research. BMC Health Services Research. 2006;6:44. doi: 10.1186/1472-6963-6-44. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 6.Vogus TJ, Sutcliffe KM. The safety organizing scale: Development and validation of a behavioral measure of safety culture in hospital nursing units. Medical Care. 2007;45(1):46–54. doi: 10.1097/01.mlr.0000244635.61178.7a. [DOI] [PubMed] [Google Scholar]
  • 7.Singer S, Kitch BT, Rao SR, Bonner A, Gaudet J, Bates DW, … Campbell EG. An exploration of safety climate in nursing homes. Journal of Patient Safety. 2012;8(3):104–124. doi: 10.1097/PTS.0b013e31824badce. [DOI] [PubMed] [Google Scholar]
  • 8.Sorra J, Carpenter J, Streagle S, Franklin M, Dyer N. Development, Pilot Testing, and Psychometric Analysis of the Nursing Home Survey on Patient Safety Culture. Agency for Healthcare Research and Quality; Rockville, MD: 2008. [Google Scholar]
  • 9.Larson EL, Cohen B, Murray M, Saiman L. Challenges in conducting research in pediatric long-term care facilities. Clin Pediatr (Phila) 2014 Oct;53(11):1041–6. doi: 10.1177/0009922814540986. [DOI] [PubMed] [Google Scholar]
  • 10.Navarra, et al. Assessing Nursing Care Needs of Children with Complex Medical Conditions: The Nursing Kids Intensity of Care Survey (N-KICS) Jl Ped Nurs. 2015 doi: 10.1016/j.pedn.2015.11.012. in presss. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 11.Caicedo C. Health and Functioning of Families of Children With Special Health Care Needs Cared for in Home Care, Long-term care, and Medical Day Care Settings. Journal of Developmental and Behavioral Pediatrics. 2015;36(5):352–61. doi: 10.1097/DBP.0000000000000167. [DOI] [PubMed] [Google Scholar]
  • 12.Friedman SL, Kalichman MA. Out-of-home placement for children and adolescents with disabilities. Pediatrics. 2014;134(4):836–46. doi: 10.1542/peds.2014-2279. [DOI] [PubMed] [Google Scholar]
  • 13.Cohen E, Kuo DZ, Agrawal R, et al. Children with Medical Complexity: An Emerging Population for Clinical and Research Initiatives. Pediatrics. 2011;127(3):529–538. doi: 10.1542/peds.2010-0910. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 14.Steering Committee on Quality Improvement and Management and Committee on Hospital Care. Principles of Pediatric Patient Safety: Reducing Harm Due to Medical Care. Pediatrics. 2011 Jun;127(6):1199–1210. doi: 10.1542/peds.2011-0967. [DOI] [PubMed] [Google Scholar]
  • 15.AHRQ. Survey User’s Guide: Nursing Home Survey on Patient Safety Culture. Agency for Healthcare Research and Quality; Rockville, MD: Oct, 2014. [Google Scholar]
  • 16.Matsunaga M. How to Factor-Analyze Your Data Right: Do’s, Don’ts, and How-To’s…. International Journal of Psychological Research. 2010;3(1):97–110. Retrieved: http://www.redalyc.org/articulo.oa?id=299023509007. [Google Scholar]
  • 17.Jackson DL, Gillaspy JA, Jr, Purc-Stephenson R. Reporting practices in confirmatory factor analysis: An overview and some recommendations. Psychological Methods. 2009;14(1):6–23. doi: 10.1037/a0014694. [DOI] [PubMed] [Google Scholar]
  • 18.Munro BH. Statistical Methods for Health Care Research. 5. Philadelphia: Lippincott; 2005. [Google Scholar]
  • 19.Costello Anna B, Osborne Jason. Best practices in exploratory factor analysis: four recommendations for getting the most from your analysis. Practical Assessment Research & Evaluation. 2005;10(7) Retrieved: http://pareonline.net/getvn.asp?v=10&n=7. [Google Scholar]
  • 20.American Educational Research Association, American Psychological Association and National Council on Measurement and Education. Standards for educational and psychological testing. 2014 [Google Scholar]
  • 21.Albright Jeremy J, Park Hun Myoung. Working Paper. The University Information Technology Services (UITS) Center for Statistical and Mathematical Computing, Indiana University; 2009. Confirmatory Factor Analysis Using Amos, LISREL, Mplus, and SAS/STAT CALIS. http://www.indiana.edu/~statmath/stat/all/cfa/index.html. [Google Scholar]
  • 22.Kaiser HF. An index of factorial simplicity. Psychometrika. 1974;39:31–36. [Google Scholar]
  • 23.Hatcher L. A step-by-step approach to using the SAS(R) system for factor analysis and structural equation modeling. Cary, NC: SAS Institute; 1994. [Google Scholar]
  • 24.StataCorp. Statistical Software. College Station, TX: StataCorp LP. Stata Press; College Station, TX: 2013. STATA Structural equation modeling reference manual, Release 13. [Google Scholar]
  • 25.Waltz CF, Strickland OL, Lenz ER, editors. Measurement in nursing and health research. 3. New York: Springer; 2005. [Google Scholar]
  • 26.Gorsuch RL. Factor Analysis. 2. Hillsdale, NJ: Lawrence Erlbaum Associates; 1983. [Google Scholar]
  • 27.Hedskold M, Pukk-Harenstam K, Berg E, Lindh M, Soop M, Ovretveit J, et al. Psychometric properties of the hospital survey on patient safety culture, HSOPSC, applied on a large Swedish health care sample. BMC Health Services Research. 2013;13(1):332. doi: 10.1186/1472-6963-13-332. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 28.Smits M, Christiaans-Dingelhoff I, Wagner C, Wal G, Groenewegen P. The psychometric properties of the ‘Hospital Survey on Patient Safety Culture’ in Dutch hospitals. BMC Health Services Research. 2008;8(1):230. doi: 10.1186/1472-6963-8-230. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 29.Bodur S, Filiz E. Validity and reliability of Turkish version of “Hospital Survey on Patient Safety Culture” and perception of patient safety in public hospitals in Turkey. BMC Health Services Research. 2010;10(1):28. doi: 10.1186/1472-6963-10-28. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 30.Zúñiga F, Schwappach D, De Geest S, Schwendimann R. Psychometric properties of the Swiss version of the Nursing Home Survey on Patient Safety Culture. Safety Science. 2013;55(0):88–118. [Google Scholar]

RESOURCES