Psychometric Evaluation of an Instrument for Measuring Organizational Climate for Quality: Evidence from a National Sample of Infection Preventionists

Abstract

In recent years, there has been increased interest in measuring the climate for infection prevention; however, reliable and valid instruments are lacking. This study tested the psychometric properties of the Leading a Culture of Quality for Infection Prevention (LCQ-IP) instrument measuring the infection prevention climate in a sample of 972 Infection Preventionists from acute care hospitals. An exploratory principal component analysis showed that the instrument had structural validity and captured four factors related to the climate for infection prevention: psychological safety, prioritization of quality, supportive work environment and improvement orientation. LCQ-IP exhibited excellent internal consistency with Cronbach’s α of 0.926. Criterion validity was supported with overall LCQ-IP scores increasing with the number of evidence-based prevention policies in place (p = 0.047). This psychometrically sound instrument may be helpful to researchers and providers in assessing climate for quality related to infection prevention.

INTRODUCTION

In recent years, there has been increased recognition of the need to improve the quality of care received by patients in acute care settings.¹ The Institute of Medicine (IOM) published its two reports on quality, To Err is Human and Crossing the Quality Chasm, highlighting the frequent occurrence and dire consequences of medical errors and outlining a comprehensive strategy to improve the delivery of care.^{1, 2} Healthcare-associated infections were recognized as one of the important quality problems by the IOM as these infections result in significant morbidity and mortality, increased length of stay and added costs of care.^{3, 4} In the last decade, there has been an increase in evidence showing that certain infections can be prevented through the use of care bundles.^{5, 6} Although several studies have shown that implementation of and adherence to evidence-based bundles is associated with lower rates of infections,^{7, 8} variation exists in the presence and compliance with these policies in U.S. hospitals.^{9, 10}

Possible causes of variation are organizational factors such as leadership, work satisfaction, and cooperation that are important components in ensuring compliance with guidelines and quality of care.^11–14 However, no studies have specifically examined the relationship between organization climate for quality defined as members’ shared perceptions that the organization expects, supports and rewards efforts to provide quality care^{15, 16} and compliance with infection prevention bundles. The absence of such studies is likely related to the absence of validated instruments for assessing organizational climate for infection prevention.

Although several validated tools exist for measuring various types of organizational constructs¹⁷ and quality-oriented climate,¹⁸ one drawback of existing instruments is the lack of specificity in measuring climate around infection prevention. Survey/quality improvement experts recommend that climate instruments be as specific and targeted as possible, in order to facilitate the identification of meaningful relationships among variables and concrete action based on survey results. Additionally, existing climate instruments are often too long to be incorporated in broader surveys intended to capture multiple constructs. For example, the Safety Attitudes Questionnaire is made up of 40 items and measures attitudes about six patient-safety related domains.¹⁸ Incorporating lengthy instruments within another survey is likely to create undue burden on survey respondents, leading to low response rates. Ideally, psychometrically valid, shorter instruments can be incorporated to allow researchers and practitioners to asses multiple variables at once without causing survey fatigue in respondents. One instrument that specifically focuses on quality-oriented climate, is relatively short, and can be adapted to assess quality with respect to a specific problem is the Leading a Culture of Quality (LCQ). However, this instrument has yet to be psychometrically evaluated. The objective of this study was to evaluate the psychometric properties of the LCQ in a national sample of infection control directors working in acute care hospitals across the country; specifically we evaluated the LCQ’s psychometric validity when assessing infection prevention climate.

BACKGROUND ON THE LCQ SURVEY

The LCQ was originally co-developed by the Institute for Clinical Systems Improvement and Satisfaction Performance Research in Minnesota, the former being an organization that consisted of 35 medical groups that wished to assess their quality-oriented climate using a relatively short and easy to administer survey and the latter being a survey research firm [P. Jury, personal communication, September 7, 2011]. The original LCQ consists of 27 items organized into nine subscales: alignment (4 items), quality focus (4 items), change orientation (3 items), change actions (2 items), openness (3 items), psychological safety (4 items), accountability (2 items), work group cooperation and respect (3 items) and workload (2 items). The items organized by original subscale are listed in Appendix I. Responses to all but one item are indicated on a Likert scale of 1 – 5, where 1 corresponds to strongly agree, and 5, strongly disagree; the responses to one item (Item 22) range from 1, never to 5, very often.

With the exception of the psychological safety subscale, which was adopted from an existing survey,¹⁹ the subscales were constructed by the survey developers. Both content and face validity of the LCQ were previously established via an expert panel and qualitative interviews conducted by survey developers. Since then, the instrument has been used with multiple personnel types with up to 20,000 surveys administered over the past seven years [P. Jury, personal communication, September 7, 2011]. The LCQ has primarily been used by organizations for self-assessment of their quality-oriented climate. Recently, it has also used by researchers to study the effects of interventions aimed to improving quality-oriented climate²⁰ and the effects of such climate on organizational outcomes.²¹ Despite this use, no published psychometric studies of the instrument were available. We conducted a psychometric analysis of a modified LCQ instrument, in which the wording was changed slightly to make it more specific to infection prevention (e.g. “quality” changed to “infection prevention”).

METHODS

Sample and Data Collection Procedures

The modified LCQ in infection prevention (LCQ-IP) instrument was embedded in a national, web-based survey of infection control directors from hospitals participating in the National Healthcare Safety Network. Infection control directors or, in the absence of a director, the person in charge of infection control at each hospital, were asked to serve as an informant for their hospital. These directors are a good population to survey regarding the infection prevention climate because their primary role involves coordinating the hospital’s efforts to improve the quality of patient care by implementing evidence based practices to prevent and control infections. Data were collected in the winter of 2011 using a modified Dillman technique for recruitment, in which an initial invitation letter was followed by weekly reminders and a final chance letter.²² The survey and the recruitment method are described in more detail elsewhere.⁹ This study was approved by the Institutional Review Boards of Columbia University Medical Center and RAND Corporation.

A total of 1,013 surveys were collected (response rate of 29% from the overall survey)⁹ with 972 participants providing complete responses to the LCQ-IP instrument. Table 1 provides demographic data of the informants’ hospitals. The largest proportion of hospitals were located in a rural setting (41%), followed by suburban (32%) and urban (26%). The average bed size of participating facilities was 239 (SD +/− 206, range 13 – 1614). The majority of hospitals (77%) were non-profit and one third (37%) were affiliated with a medical school. A comparison of the study sample to the non-responding hospitals showed that the facilities participating in the study were larger; however, there were no differences between the respondents and non-respondents in terms of medical school affiliation, ownership and most notably infection rates.⁹

Table 1.

Characteristics of Study Hospitals, N = 972

		N	%
Affiliated with Medical School*
	Yes	259	37
	No	440	63
Ownership Status*
	Non-Profit	535	77
	Private	122	17
	Other	42	6
Facility part of a larger system that shares Infection Prevention resources
	Yes	285	29
	No	674	69
	Missing	13	1
Participation in Infection Control Initiative
	Yes	626	64
	No	323	33
	Missing	23	2
Location
	Urban Setting	253	26
	Suburb	314	32
	Rural Setting	399	41
	Missing	6	1
Region
	Northeast	180	19
	Midwest	263	27
	South	342	35
	West	164	17
	Missing	23	2
Infection Prevention Program has an Infection Control Director position
	Yes	690	71
	No	281	29
	Missing	1	0
		Mean (SD)
Beds		239 (+/−206)

% may not up to 100% due to rounding.

^*Based on an N of 699

Data Analysis

Only hospitals with complete survey data (N = 972) were included in our analyses, which were conducted using SPSS Version 22.0 (SPSS Inc, Chicago, IL). In the first stage of our analysis, we prepared and evaluated the individual items. Specifically, we reverse-coded two negatively worded items (Items 23 and 24, See Appendix I) such that a lower score corresponded to a negative response (e.g. strongly disagree). Additionally, descriptive statistics for each item were examined including mean and standard deviation as well as the correlation matrix. Inter-item correlations were examined to identify highly correlated items (i.e., items with a correlation of .70 or higher). Highly correlated items were deleted to eliminate redundancy and improve factor structure.²³

Our psychometric analysis focused on assessing 3 core properties of the LCQ-IP instrument: structural validity (the degree to which the instrument adequately reflects the dimensionality of the construct), internal consistency (the reliability of the embedded subscales) and criterion validity (the ability of the instrument to estimate or predict the values of other related measures or effects). Each of these is regarded as critical to assessing the psychometric strength of an instrument; a good instrument will perform well with respect to each property.

Structural validity

We conducted factor analysis to assess the structural validity of the LCQ-IP. Based on recommendations for sample size, with 27 items this study minimally required 270 subjects.²⁴ Thus, we had an adequate sample size. The Kaiser-Meyer-Olkin Measure of Sampling Adequacy (KMO test) and Bartlett’s Test of Sphericity were used to assess the appropriateness of the overall factor analysis. The anti-image correlation matrix was examined to further assess if the correlation matrix was factorable with values of ≥ .90 indicating ‘marvelous’ measures of sampling adequacy.²⁵ An exploratory principal components analysis (PCA) with varimax rotation was conducted to consolidate items and identify unique factors. The PCA method was selected presuming no a priori hypothesis about factor structure. The specific criteria that determined the number of factors and the number of items within a factor included the point of discontinuity of the scree plot and eigenvalues greater than 1. Once the number of factors being extracted was determined, varimax rotation was conducted to simplify the factor structure and item factor loadings and eigenvalues were examined. The initial eigenvalues were examined to identify the amount of variance explained by each factor, and cumulatively. Items were assigned to a factor if the loading was greater than 0.40. Items with factor loadings of 0.40 or higher on multiple factors indicating a complex structure were deleted if the difference between the loadings was less than 0.15. Additionally, items with factor loadings of less than 0.40 on all factors were eliminated.

Internal consistency

The internal consistencies of the final reduced LCQ-IP and each subscale were evaluated using Cronbach’s α coefficients. Consistent with existing guidelines, scales with internal consistencies of greater than or equal to 0.70 were considered acceptable.²⁶

Criterion validity

To assess criterion validity, the association between the overall LCQ-IP instrument and the number (range 0 to 5) of evidence based policies for prevention of central-line associated bloodstream infections (CLABSI) in place⁹ was assessed using analysis of variance (ANOVA). We hypothesized hospitals with more positive climates toward infection prevention would have more infection prevention evidence-based policies in place.

RESULTS

Assessment of the correlation matrix (not shown) indicated two pairs of highly correlated items. Item 14, “I observe a high level of cooperation among all members of my work unit or department” was highly correlated (r = 0.777) with item 15, “There is a climate of trust in my department or work unit.” Item 18, “My organization’s senior leadership has focused the organization in the right direction” was highly correlated with item 19, “I am satisfied with the information I receive from management on what’s going on in the organization” (r = 0.736). In addition, item 18 was also correlated with item 16 (r = 0.696), “I have a clear understanding of the organization's mission, vision, and values.” Based on these results, items 15 and 18 were removed from further analysis in order to improve the factor structure. We conducted a factor analysis on the 25 remaining items.

Our analysis indicated that the LCQ-IP showed structural validity, as the instrument captured factors related to a climate for infection prevention. The KMO test yielded a value of 0.959 and the p value for the Bartlett’s test was <0.001, indicating that the data could be factor analyzed. Additionally, an examination of the anti-image correlation matrix for the individual items showed that the KMO measure of sampling adequacy was greater than 0.9, further supporting the use of PCA. The PCA resulted in a four-factor solution (Table 2) that explained a total of 58.8% of the variance. One item (# 21) was deleted due to low factor loadings and five items (# 1, # 14, # 16, # 19 and # 24) were eliminated due to high loading on multiple factors leaving 19 items across four factors. Factor 1 consisted of seven items explaining 18.2% of the variance. Items loading on this factor reflected the respondents’ perception that employees are respected and can speak freely without the fear of repercussions; therefore, this factor was named “Psychological Safety.” Factor 2 consisted of five items reflecting the extent to which an emphasis on quality care permeates the organization’s mission and action and was named “Prioritization of Quality.” The next factor included four items that focused on whether leaders and organizational work policies enabled infection prevention and was named “Supportive Work Environment.” Finally, Factor 4 included three items that reflected the organization’s improvement oriented environment; this factor was named “Improvement Orientation.”

Table 2.

Items, means and factor loadings (N = 972)

Item #	Mean	SD	Factor Loadings
			1	2	3	4
Psychological Safety
8. The climate in the organization promotes the free exchange of ideas.	3.93	.812	.620
9. Staff will freely speak up if they see something that may improve patient care or affect patient safety.	3.92	.829	.624
10. I feel free to express my opinion without worrying about the outcome.	4.02	.874	.547
13. In general, people in our organization treat each other with respect.	4.16	.691	.599
25. People in this organization are comfortable checking with each other if they have questions about the right way to do something.	4.02	.656	.723
26. The people in this organization value others’ unique skills and talents.	3.90	.701	.744
27. Members of this organization are able to bring up problems and tough issues.	3.83	.761	.723
Prioritization of Quality
2. The HAI prevention goals and strategic plan of our organization are clear and well communicated.	4.15	.761		.645
3. Results of our infection prevention efforts are measured and communicated regularly to staff.	4.17	.804		.719
4. There is a good information flow among departments to provide high quality patient safety and care.	3.91	.823		.666
6. People here feel a sense of urgency about preventing healthcare associated infections.	3.65	.906		.624
7. Employees are encouraged to become involved in infection prevention.	4.17	.757		.673
Supportive Work Environment
5. Senior leadership here has created an environment that enables changes to be made.	3.90	.890			.532
20. Where I work, people are held accountable for the results of their work.	3.61	.999			.425
22. The quality of work suffers because of the amount of work suffers because of the amount of work staff are expected to do.	3.08	.833			.769
23. Most people in this organization are so busy that they have very little time to devote to infection prevention efforts.*	3.12	.994			.729
Improvement Orientation
11. I can think of examples when problems with patient infections have led to changes in our procedures or equipment.	4.32	.689				.638
12. I know of one or more healthcare associated infection prevention initiatives going on within our organization this year.	4.53	.605				.732
17. I have a clear understanding of the organization’s mission, vision and values.	4.43	.637				.707
% Variance Explained			18.2	13.8	13.5	13.3

Extraction Method: Principal Component Analysis. Rotation Method: Varimax with Kaiser Normalization.

^*Reverse-coded

The mean scores for the individual items within factors (Table 2) as well as the mean scores for each factor (Table 3) were high, indicating positive organizational climates for infection prevention overall. However, the standard deviation (20% of the mean on average) indicated variability in climates’ support for infection prevention. Additionally, there was variability in the degree to which each factor was present. The “Improvement orientation” factor received the highest mean score (mean = 4.43; S.D. = 0.52) and “Supportive Work Environment” received the lowest mean score (mean = 3.42; S.D. = 0.71).

Table 3.

Reliabilities for the LCQ-IP and Four Newly Developed Subscales

Factor	# of items	Mean (SD)	α
1: Psychological Safety	7	3.97 (0.59)	0.883
2: Prioritization of Quality	5	4.01 (0.63)	0.840
3: Supportive Work Environment	4	3.43 (0.71)	0.767
4: Improvement Orientation	3	4.43 (0.52)	0.724
Total scale	19	3.94 (0.52)	0.926

Note: LCQ-IP = Leading a Culture of Quality in Infection Prevention

The internal consistency reliabilities for the overall revised instrument and the four sub-scales (representing each of the factors) are presented in Table 3. The Cronbach’s α for each sub-scale ranged from 0.724 for Improvement Orientation (3 items) to 0.883 for Psychological Safety (7 item). The overall 19-item instrument exhibited an α of 0.926, indicating excellent internal consistency.

Table 4 provides evidence that the instrument has criterion validity also. The mean LCQ-IP scores increased with the studied criterion: the number of CLABSI policies in place (p = 0.047).

Table 4.

Relationship Between Presence of CLABSI Policies and LCQ-IP

# of CLABSI Policies	N	Total Climate Score (Mean)	SD
0	19	69.5	8.7
1	4	67.3	15.4
2	8	72.9	9.1
3	26	73.1	10.4
4	105	74.0	10.3
5	702	75.3	9.7

Note: CLABSI = central line associated bloodstream infection; LCQ-IP = Leading a Culture of Quality in Infection Prevention

p-value from ANOVA = 0.047

DISCUSSION

This is the first analysis conducted to evaluate the psychometric properties of an instrument for assessing infection prevention climate, the LCQ-IP, in a national sample of hospital infection control directors. Our results suggests that the LCQ-IP is psychometrically sound in several respects, as it demonstrated structural and criterion validity as well as reliability. Thus, this instrument may be useful to others wishing to measure infection prevention climate. The instrument may also be useful to those wishing to adapt it to measure other quality-related climates such as patient safety or to those interested in quality-oriented climate generally.

The principal component analysis resulted in a reduced instrument consisting of 19 items and the identification of four factors indicating that the LCQ consists of fewer distinct concepts than originally conceptualized. The four factors include some of the constructs found in the original version of the survey and those found in other instruments measuring quality-oriented climate.¹⁸ The four items making up the “Supportive Work Environment” factor describe the perceived work environment of the respondent and come from multiple subscales in the original instrument including workload (2 items), accountability (1 item) and change orientation (1 item). The “Prioritization of Quality” items came from the quality focus and change orientation subscales of the original instrument. Nembhard et al. also found that combining these subscales results in a single, reliable scale.²⁰

This study has a number of strengths and limitations. First, this was the first study to psychometrically test the LCQ-IP in a large, national sample of infection control directors. While only one employee from each institution completed the survey, the sample was homogeneous as participants had similar roles within their respective institutions. However, this may limit the reliability of the results and prevents us from assessing climate as a shared perception. Therefore, we recommend future psychometric analyses in other samples (i.e. physicians, allied health professionals), with multiple respondents and with assessment of test- retest reliability. This is especially important since previous researchers have found that staff report climate differently based on their profession.²⁰ Last, while we examined criterion validity based on the presence of evidence-based guidelines and previously it has been found that the presence of these guidelines are associated with lower infection rates; we were not able to test the predictive validity of the LCQ-IP using infection rates.

CONCLUSION

Examining the organizational climate, particularly around infection prevention, has become a priority in healthcare. This study contributes to the field by evaluating the psychometric properties of an instrument that might be used to facilitate the examination: the LCQ-IP. We found that the LCQ-IP captures core dimensions of an infection prevention climate and performs well on several psychometric measures used to assess the quality of an instrument. Thus, the LCQ-IP may be helpful tool for researchers and healthcare providers aiming to assess hospital’s climate for quality specifically related to infection prevention and control. Furthermore, this instrument may be modified and useful in assessing other quality-related climate.

Appendix 1. Leading a Culture of Quality (LCQ) Measure

Subscale	General Description	Number of Items
Quality Focus	Leaders demonstrating their commitment to quality and continuous improvement. Good flow of communication across departments to ensure high quality patient care and safety.	1.Senior management shows by its action that preventing HAI is a top priority in this organization. 2.The HAI prevention goals and strategic plan of our organization are clear and well communicated. 3.Results of our infection prevention efforts are measured and communicated regularly to staff. 4.There is a good information flow among departments to provide high quality patient safety and care.
Change Orientation	The organization's appetite and commitment to constantly identify and implement necessary changes. Leadership creating an environment that enables changes to be made.	5.Senior leadership here has created an environment that enables changes to be made. 6.People here feel a sense of urgency about preventing HAI. 7.Employees are encouraged to become involved in infection prevention.
Openness	Relying on the inputs of people at all levels in the organization. People being willing to speak their mind and exchange ideas freely.	8.The climate in the organization promotes the free exchange of ideas. 9.Staff will freely speak up of they see something that may improve patient care or affect patient safety. 10.I feel free to express my opinion without worrying about the outcome.
Change Actions	When employees can point to real and meaningful examples of improvement. The visible affirmation of alignment between leadership's words and actions.	11.I can think of examples when problems with patient infections have led to changes in our procedures or equipment. 12.I know of one or more HAI prevention initiatives going on within our organization this year.
Work Group Cooperation and Respect	When a culture is founded on a climate of trust, people will be open to taking personal risk and working together for change.	13.In general, people in our organization treat each other with respect. 14.I observe a high level of cooperation among all members of my work unit or department. 15.There is a climate of trust in my department or work unit.
Alignment (with leadership and direction)	Understanding where the organization is headed and why. Understanding and embracing the organization's mission, vision, and values.	16.My organization is making the changes necessary to compete effectively. 17.I have a clear understanding of the organization’s mission, vision and values. 18.My organization’s senior leadership has focused the organization in the right direction. 19.I am satisfied with the information I receive from management on what’s going on in the organization.
Accountability	In times of change, people receive regular feedback on how they are performing on quality and continuous improvements. Each person faithfully carry's out the necessary changes in their work.	20.Where I work, people are held accountable for the results of their work. 21.I receive regular ongoing feedback about my job performance.
Workload	The organization's ability to maintain quality and make improvements without overwhelming people. Continually examining the work processes and the organizational staffing priorities to successfully integrate quality and continuous improvement into their daily work lives.	22.The quality of work suffers because of the amount of work staff are expected to do. 23.Most people in this organization are so busy that they have very little time to devote to infection prevention efforts.
Psychological Safety	A climate in which it is safe to speak up with questions, concerns, and suggestions. The focus is on productive conversations that enable early prevention of problems and achievement of shared goals, because people are less focused on self-protection.	24.If you make a mistake in this organization, it tends to be held against you. 25.People in this organization are comfortable checking with each other if they have questions about the right way to do something. 26.The people in this organization value others’ unique skills and talents. 27.Members of this organization are able to bring up problems and tough issues.
		Total: 27

Adapted from P. Jury (personal communication, September 7, 2011)