Skip to main content
PMC home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2017 Mar 1.
Published in final edited form as: Child Abuse Negl. 2015 Nov 10;53:17–26. doi: 10.1016/j.chiabu.2015.10.017

Validating the Implementation Climate Scale (ICS) in Child Welfare Organizations

Mark G Ehrhart 1,§, Elisa M Torres 2,3, Lisa A Wright 1, Sandra Y Martinez 1, Gregory A Aarons 2,3
PMCID: PMC4818155  NIHMSID: NIHMS736925  PMID: 26563643

Abstract

There is increasing emphasis on the use of evidence-based practices (EBPs) in child welfare settings and growing recognition of the importance of the organizational environment, and the organization’s climate in particular, for how employees perceive and support EBP implementation. Recently, Ehrhart, Aarons, and Farahnak (2014) reported on the development and validation of a measure of EBP implementation climate, the Implementation Climate Scale (ICS), in a sample of mental health clinicians. The ICS consists of 18 items and measures six critical dimensions of implementation climate: focus on EBP, educational support for EBP, recognition for EBP, rewards for EBP, selection or EBP, and selection for openness. The goal of the current study is to extend this work by providing evidence for the factor structure, reliability, and validity of the ICS in a sample of child welfare service providers. Survey data were collected from 215 child welfare providers across three states, 12 organizations, and 43 teams. Confirmatory factor analysis demonstrated good fit to the six-factor model and the alpha reliabilities for the overall measure and its subscales was acceptable. In addition, there was general support for the invariance of the factor structure across the child welfare and mental health sectors. In conclusion, this study provides evidence for the factor structure, reliability, and validity of the ICS measure for use in child welfare service organizations.

Keywords: Implementation climate, evidence-based practice, organizational climate, organizational context, measurement

There is increasing awareness of the importance of evidence-based practices (EBPs) for treating clients in public health and allied health sectors, particularly in the child welfare sector. Although progress has been made, there are still serious gaps in what is known about how to improve practice and successfully implement and sustain EBPs in community agencies (Pecora, Whittaker, Malluccio, & Barth, 2012; Horwitz, Chamberlain, Landsverk, & Mullican, 2010; Proctor, et al., 2009), leading to a discrepancy between the practices that research has concluded are effective (i.e., EBPs) and the practices that are actually being implemented in public health settings (Aarons, Ehrhart, Farahnak, & Sklar, 2014; Garland, Hurlburt, & Hawley, 2006; Lobb & Colditz, 2013; Proctor et al., 2009). Several factors contribute to this discrepancy, including insufficient EBP communication, training, time, resources, infrastructure, environmental constraints, and systems to support the use of EBPs (Aarons & Palinkas, 2007; Kessler, Gira, & Poertner, 2005). Thus, researchers agree that EBP adoption in public health settings occurs too slowly (Mitchell, 2011; Stirman, Crits-Christoph, & DeRubeis, 2004), and that for long-term sustainment of EBPs, more knowledge on the various factors that either facilitate or impede EBP implementation is needed (Aarons, Ehrhart, & Farahnak, 2014; Garland et al., 2006; Mitchell, 2011).

There have been a number of calls for more application of implementation science (“the study of methods to promote the integration of research findings and evidence into healthcare policy and practice;” Fogarty International Center, 2013) in child welfare. For example, efforts have been made to better understand and overcome barriers and strengthen facilitators to implementation and sustainment by developing strategies to promote adoption and use of EBPs (Aarons, Hulburt, & Horwitz, 2011; Kaye, DePanfilis, Bright, & Fisher, 2012; Mildon & Shlonsky, 2011). Some service systems and organizations have used implementation frameworks and approaches to support implementation processes (Aarons, et al., 2012; Barbee, Christensen, Antle, Wandersman, & Cahn, 2011; Pipkin, Sterrett, Antle, & Christensen, 2013). There are also examples of the use of collaborative approaches (Aarons, et al., 2014; Kolko, et al., 2012) as well as consideration of costs (Holmes, et al., 2014) and organizational social context (Glisson, Green, & Williams, 2012) during the implementation process. For the present study, we focus on organizational context, and specifically on context for EBP implementation.

Organizational social context has been found to impact the adoption and implementation of EBPs (Aarons, et al., 2012). Although several organizational factors have been identified as important for implementing EBPs successfully (e.g., culture, leadership, and network connectedness; Mitchell, 2011), researchers have begun to emphasize the critical role that organizational climate plays in how employees perceive and support EBP implementation. Most of this research has focused on molar organizational climate, which captures broad factors that describe employees’ general experiences in their organization (Ehrhart et al., 2014). Researchers found that molar organizational climate is related to attitudes, motivation, and performance, all of which are key for implementation success (Aarons, Ehrhart, Farahnak, & Sklar, 2014; Glisson & James, 2002; Kuenzi & Schminke, 2009). Researchers also found that when the climate within the organization was more positive and engaged, child welfare outcomes were improved (Glisson & Green, 2011; Glisson & Hemmelgarn, 1998; Williams & Glisson, 2014). Additionally, organizational climate has been linked to employee willingness to use EBPs (Aarons, Green, et al., 2012). Although molar organizational climate provides useful information to help facilitate implementation, recent advances in organizational science suggest that climates specifically focused on the criterion of interest may have added benefits for understanding the role of the organizational context in achieving strategic goals (Ehrhart, Schneider, & Macey, 2014). The most well-developed areas of research on focused climate include climate for customer service (e.g., Schneider, Ehrhart, Mayer, Saltz, & Niles-Jolly, 2005) and climate for safety (e.g., Zohar & Luria, 2005), but there has also been growing interest in recent years in the application of the focused climates to a number of organizational strategic goals and processes, including implementation (Klein, Conn, & Sorra, 2001) and specifically EBP implementation (Aarons, Ehrhart, Farahnak, & Sklar, 2014; Ehrhart et al., 2014; Jacobs, Weiner, & Bunger, 2014; Weiner, Belden, Bergmire, & Johnston, 2011).

Climate for EBP implementation is defined as “employees’ shared perceptions of the importance of EBP implementation within the organization” (Ehrhart, Aarons, & Farahnak, 2014). Implementation climate has been identified as both measurable and an important strategic climate in organizations (Aarons, Ehrhart, Farahnak, & Sklar, 2014; Jacobs et al., 2014; Weiner et al., 2011). An implementation climate develops when leaders and the organization emphasize the importance of implementing EBPs (Aarons, Ehrhart, Farahnak, & Sklar, 2014, Ehrhart, Aarons, & Farahnak, 2014; Weiner et al., 2011). A successful EBP implementation climate “captures the extent to which employees perceive that the adoption, implementation, and use of an innovation such as EBP is expected, rewarded, and supported by the organization” (Ehrhart, Aarons, & Farahnak, 2014, p. 2). Note that such a strategic climate is distinct from both molar climate, which captures employee perceptions of the general work environment, and organizational culture, which typically focuses on the deeper-level assumptions in the organization, captures a broader range of facets of organizational behavior (from assumptions to espoused values to artifacts; Schein, 2010), is more difficult to change, and typically does not have a strategic focus (Ehrhart, Schneider, & Macey, 2014). When implementation climate is high in the organization, employees perceive the organization values EBPs and is supportive in practicing them. It has been proposed that such perceptions lead to increased levels of attention and motivation for EBP implementation, ultimately enhancing EBP outcomes (Aarons, Ehrhart, Farahnak, & Sklar, 2014; Ehrhart Aarons, & Farahnak, 2014; Jacobs, Weiner, Reeve, Hofmann, Christian, & Weinberger, 2015; Weiner et al., 2011).

Due to the multiple mechanisms that can contribute to an employee’s perception of what an organization values, a multidimensional measure of implementation climate is necessary to incorporate all aspects of effective EBP implementation. In an effort to answer these calls and develop a measure of climate for EBP implementation and its specific dimensions, Ehrhart, Aarons, and Farahnak (2014) developed the Implementation Climate Scale (ICS). This multidimensional measure of EBP implementation considers a variety of factors that contribute to successful implementation. Items for the ICS were developed based on EBP implementation and strategic climate literature, and an exploratory factor analysis revealed six dimensions of EBP implementation (i.e., selection for openness, recognition for EBP, selection for EBP, focus on EBP, education support for EBP, and rewards for EBP). The ICS was validated via exploratory and confirmatory factor analyses and demonstrated good psychometric properties in the mental health setting (see Ehrhart, Aarons, & Farahnak, 2014, for details).

As identified in the Exploration, Preparation, Implementation, sustainment (EPIS) implementation framework (Aarons, Hurlburt, & Horwitz, 2011), child welfare settings share many similarities with mental health settings in regard to system and organizational level issues such as leadership, contracting, and service funding. However, child welfare organizations also have a number of distinct attributes due to the structure of the system (e.g., policy and legislative mandates) as well as the processes (e.g., case management, provision of in-home services) and service population of child welfare (Pecora, Whittaker, Maluccio, & Barth, 2012). For example, because service providers must account for both the client (i.e., the child) and the client’s direct caregivers, this creates a unique challenge in child welfare that may be qualitatively different than parent engagement in mental health settings or other allied health care settings. Because of these differences, several obstacles to effective EBP implementation as it relates specifically to child welfare have been identified, such as the complex environment of decision-making in child welfare (e.g., judges as the final authority in making decisions; Kessler, et al., 2005). In considering these commonalities and differences, the ICS was developed to be applicable across multiple settings as suggested by the EPIS framework.

In summary, the present study objective is to expand research on EBP implementation climate by confirming the six-factor measurement model of the ICS in a sample of child welfare providers. Validating the implementation climate measure within the child welfare setting will further support the generalizability of the ICS and provide a useful tool that can be utilized to improve child welfare practice. It is also in line with the recent push by researchers in implementation science for the development and evaluation of measures related to EBP implementation (Rabin et al., 2012; Proctor, Powell, & Feely, 2014). We hypothesized that the six-factor structure of the ICS would be validated in the child welfare setting, with items loading on the hypothesized factors of selection for openness, recognition for EBP, selection for EBP, focus on EBP, educational support for EBP, and rewards for EBP. In addition, as an exploratory analysis we also evaluated whether the ICS was invariant across both child welfare and mental health providers should researchers seek to make direct comparisons between these contexts.

Method

Participants

The sample consisted of 215 direct service providers working for 12 different community-based organizations contracted by child welfare systems in California, Illinois, and Oklahoma. Organizations were chosen for participation based on previously established relationships with organizational leadership. Of the 230 employees who were eligible to participate, 215 service providers forming 43 teams agreed to participate (response rate = 93.5%; 120 were from California, 13 were from Illinois, and 82 were from Oklahoma). The eligibility criteria were that they provide direct services to clients and that they worked with their team for at least six weeks. Teams were determined based on the individual’s direct supervisor (i.e., the supervisor who completes service provider’s performance evaluation); each team was comprised of individuals all reporting to the same supervisor. The average team size was 5.0 (SD = 3.0, range = 1–16).

Overall, the sample was 92.6% (N = 199) female and 7.4% (N = 16) male with an average age of 39.6 years (SD = 10.1, range 23 – 72). The participants’ demographic information is summarized in Table 1, and contains information on participant race, ethnicity, education, tenure, and position. Note that compared to the national sample of caseworkers in the National Survey of Child and Adolescent Well-Being (NSCAW II; Dolan, Smith, Casanueva, & Ringeisen, 2011), this sample was similar in age (current study = 39.6, NSCAW II=72.1% in the 25–44 range), percent Caucasian (current study = 62%, NSCAW II=57.5%), and tenure in child welfare (current study = 6.2 years, NSCAW II=7.1 years). However, the present study had a higher percentage of female providers (female current study = 92.9%, NSCAW II=79.3%), had lower tenure at their current organization (current study = 3.4 years, NSCAW II=5.0 years), was more educated (42.8% vs. 25% with a master’s degree) had a lower proportion of African American providers (16.0% vs. 23.7%), and had a higher proportion of Hispanic providers (20.9% vs. 14.9%).

Table 1.

Participant information.

Characteristics Values
Race
  Caucasian 62.0%
  African-American 16.0%
  Asian-American 3.8%
  Native American 5.6%
  “Other” 12.7%
Ethnicity
  Hispanic 20.9%
  Non-Hispanic 79.1%
Education
  No college 0.9%
  Some college 6.0%
  College degree 38.1%
  Master’s degree 42.8%
  Ph.D. or M.D. 0.9%
Gender
  Female 92.9%
  Male 7.4%
Position
  Intern/trainee 6.1%
  Licensed provider 37.4%
  Not an intern, but also not a licensed provider 56.5%
Age
  Mean (SD) 39.6 (10.1)
Tenure with organization
  Mean (SD) 3.4 (3.7)
Tenure in child welfare
  Mean (SD) 6.2 (6.1)
Tenure in team/with supervisor
  Mean (SD) 2.0 (2.4)
Open in a new tab

The study was approved by the appropriate Institutional Review Boards (San Diego State University and University of California, San Diego), and participants gave consent to participate. Participation was voluntary, and participants were informed they could withdraw from participating in the study at any time without negative consequences. For the measurement invariance analysis, the results from the child welfare sample were compared with those in Ehrhart, Aarons, and Farahnak’s (2014) mental health sample; details on that sample can be found in the original paper.

Procedure

Recruitment for the study began with the research team approaching CBO executives, providing them with a short description of the study. Researchers coordinated with the executives to gain permission to recruit employees. Interested organizations granted the research team permission to recruit their employees, and provided the research team with the organizational charts and contact information. Employees were then contacted to participate in the study. For the majority of the participants (93%), data were collected via online surveys, and employees received a $15 gift certificate to an online retailer as an incentive. The staff survey took approximately 20–30 minutes to complete. Each participant was emailed a unique password and username, in addition to the link to the survey. Supervisors who completed a longer survey that took approximately 30–40 minutes to complete, received $30 gift certificate incentives. Those who consented to participate could then access the survey questions.

In cases in which it was not practical to obtain data via an online survey (7% of the sample), in-person data collections at the organizations of interest were scheduled. The research team reserved an hour during a regularly occurring team meeting, and passed out surveys to all eligible participants. Participants received the same incentive as was described for online data collections. To ease concerns regarding confidentially, supervisors completed the survey in a room separate from participants. If participants were not able to complete the survey in-person and collecting data online was not practical, survey were mailed to the participating organizations for distribution to eligible employees. Employees were provided prepaid envelopes to send completed surveys back to the research team. Note that there were no significant differences in the ICS measure or its subscales based on the method of survey completion.

Measures

To measure implementation climate, the Implementation Climate Scale (ICS; Ehrhart, Aarons, & Farahnak, 2014) was utilized. This measure had originally been developed for an NIMH measure development grant (R21MH098124, PI: Ehrhart) to evaluate the extent to which the organization contains a strategic climate supportive of EBP implementation. The validated scale contains 18 items on a 5-point scale ranging from 0 (not at all) to 4 (very great extent). A full list of ICS items can be found in the supplementary materials included with the Ehrhart, Aarons, & Farahnak (2014) article. In this study, the referent for the scale was the program, which was determined based on the service provider’s direct supervisor and was the common term used to refer to the work group in child welfare CBOs (based on subject matter expert feedback).

Statistical Analyses

Confirmatory factor analysis (CFA) was conducting using Mplus statistical software accounting for the nested data structure within teams (TYPE=COMPLEX) and using maximum likelihood estimation with robust standard errors (MLR) to adjust the standard error and chi-square values for non-normality (Muthén & Muthén, 1998–2014). Any missing data were entered using full information maximum likelihood (FIML) estimation. To determine model fit, several fit indices were utilized: comparative fit index (CFI;), with values greater than .95 indicating good fit; the root mean square error of approximation (RMSEA), with values less than .06 indicating good fit, and the standardized root mean square residual (SRMR;), with values less than .08 indicating good fit (Hu & Bentler, 1999). Using multiple fit indices in cases in which sample sizes are large and non-normality is limited helps reduce Type II error (Guo, Chen, Wang, & Meng, 2008).

Cronbach’s alpha was used to assess the internal consistency reliability of the subscales and composite scale. Intraclass correlations (ICCs) and the average correlation within group (awg(j)) for each subscale were calculated to evaluate the aggregation of the individual-level responses to the unit level. To further establish validity, and confirm the assumption that the ICS measures the same construct in different sectors, measurement invariance was assessed. The purpose of examining measurement invariance was to determine whether the ICS could be used in both child welfare and mental health settings to make meaningful comparisons of latent constructs across sectors. Thus, the objective was to examine whether the ICS factor loadings were comparable across these two settings (i.e., weak factorial invariance; Meredith, 1993), as such evidence is required prior to making meaningful comparisons across groups (Ployhart & Oswald, 2004). The Yuan-Bentler Chi-Square difference test was conducted to evaluate measurement invariance.

Results

Confirmatory Factor Analysis

A six-factor implementation climate model was tested using maximum likelihood confirmatory factor analysis. The focus on EBP, educational support for EBP, recognition for EBP, rewards for EBP, selection for EBP, and selection for openness latent variables were each indicated by three observed variables. This model demonstrated good fit as indicated by multiple fit indicators (χ2(120) = 225.97, p < .001; CFI = .94; SRMR = .067; RMSEA = .064, 90% CI [.051–.077], Probability RMSEA ≤ .05 = .038), providing support for the factorial validity of the hypothesized measurement model. As shown in Table 2, the standardized factor loadings ranged from .50 to .96 and all were statistically significant (p’s < 0.001). Table 2 also shows the unstandardized factor loadings, which were also statistically significant (p’s < 0.001). Based on these findings, we accepted the six-factor model without additional modifications. Table 3 contains a table of the ICS subscale correlation matrix at both the individual and team levels. The large majority of the correlations were greater than .35, and for the most part, the strength and pattern of the correlations was in line with what was found in a mental health sample by Ehrhart et al. (2014). The only exception was for Rewards for EBP; the individual-level correlations for that subscale with the other subscales (average r = .23) were lower than the correlations among the other subscales (average r = .44) and also lower than the corresponding correlations in the Ehrhart et al. paper (average r = .32). The Rewards for EBP subscale was not correlated with any of the other subscales at the team level.

Table 2.

CFA factor loadings for the Implementation Climate Scale.

ICS Factor items CFA
Standardized
Factor Loadings		 CFA
Unstandardized
Factor Loadings
Focus on EBP
  1. Main goal is to use EBP effectively 0.84 1.00
  2. Think implementation is important 0.92 1.08
  3. Using EBP is a top priority 0.77 0.92
Educational Support for EBP
  4. Conferences, workshops, or seminars 0.74 0.93
  5. EBP trainings or in-services 0.89 1.00
  6. Training materials, journals, etc. 0.73 0.86
Recognition for EBP
  7. Seen as clinical expert 0.77 0.89
  8. Held in high esteem 0.92 1.00
  9. More likely to be promoted 0.50 0.61
Rewards for EBP
  10. Financial incentives for use of EBP 0.76 1.00
  11. More likely to get a bonus/raise 0.84 1.15
  12. Accumulate compensated time 0.53 0.86
Selection for EBP
  13. Previously used EBP 0.80 1.00
  14. Formal education supporting EBP 0.79 0.94
  15. Value EBP 0.91 1.10
Selection for Openness
  16. Adaptable 0.96 1.00
  17. Flexible 0.93 1.03
  18. Open to new interventions 0.56 0.70
Open in a new tab

Note: All loadings significant at p < 0.001.

Table 3.

Individual-level and team-level ICS subscale correlation matrix.

1 2 3 4 5 6
1. Focus on EBP --- .67** .56** .05 .61** .61**
2. Educational Support for EBP .55** --- .58** .19 .53** .53**
3. Recognition for EBP .39** .34** --- .25 .56** .64**
4. Rewards for EBP .14* .19* .35** --- .19 .10
5. Selection for EBP .44** .31** .42** .24** --- .74**
6. Selection for Openness .51** .37** .43** .21* .65** ---
Open in a new tab

Note: Individual-level correlations (n=212–215) are below the diagonal, and team-level correlations (n=43) are above the diagonal.

*

p < 0.05

**

p < 0.001.

Scale Reliability and Aggregation Statistics

The ICS scale reliabilities, individual-level item means SDs, skewness, and kurtosis values, and the team-level aggregation statistics are shown in Table 4. Internal consistencies for the scales ranged from .73–.88, indicating good internal consistency. The overall ICS alpha reliability was .89. In order to determine the amount of dependency among observations within groups, intraclass correlations, specifically ICC(1) was calculated. The ICC(1) for the overall ICS scale was .16, which indicates a large effect for unit. The ICC(1) values for the subscales ranged from −.02 to .23; all values were .07 or larger except for the value for the Rewards for EBP subscale. In addition to calculating ICC(1), we also calculated the average agreement within group for the six scales using awg(j) (Brown & Hauenstein, 2005). Five out of the six ICS dimensions provided strong awg(j) values, ranging from .74 to .78. Rewards for EBP was the only dimension that had a low, but still acceptable awg(j) value at .62. The awg(j) value for the total ICS scale was .74. Examining the pattern of all the aggregation statistics, there is evidence to support evaluating the ICS items and dimensions as unit-level constructs. We discuss some issues specific to the Rewards for EBP subscale in the Discussion section.

Table 4.

Summary statistics for the ICS total scale and subscales.

Mean SD Min Max Skewness Kurtosis α ICC(1) awg(j)
Implementation Climate Scale total 2.23 .66 0.56 4.00 −.19 .20 .89 .16 .74
Implementation Climate subscales and items
  Focus on EBP 2.96 .90 0.33 4.00 −.73 −.12 .88 .23 .74
    1. Main goal is to use EBP effectively 3.07 1.00 0.00 4.00 −1.00 .45
    2. Think implementation is important 2.90 .99 0.00 4.00 −.79 .13
    3. Using EBP is a top priority 2.92 1.01 0.00 4.00 −.89 .44
  Educational Support for EBP 2.62 1.01 0.00 4.00 −.62 −.12 .82 .12 .75
    4. Conferences, workshops, or seminars 2.45 1.25 0.00 4.00 −.50 −.72
    5. EBP trainings or in-services 2.88 1.10 0.00 4.00 −..82 −.10
    6. Training materials, journals, etc. 2.54 1.16 0.00 4.00 −.46 −.64
  Recognition for EBP 1.89 1.00 0.00 4.00 .09 −.34 .77 .07 .74
    7. Seen as clinical experts 2.01 1.20 0.00 4.00 −.17 −.84
    8. Held in high esteem 2.43 1.12 0.00 4.00 −.43 −.53
    9. More likely to be promoted 1.24 1.28 0.00 4.00 .77 −.46
  Rewards for EBP .82 .97 0.00 4.00 1.40 1.91 .73 −.02 .62
    10. Financial incentives for use of EBP .61 1.10 0.00 4.00 1.81 2.42
    11. More likely to get a bonus/raise .75 1.14 0.00 4.00 1.53 1.53
    12. Accumulate compensated time 1.11 1.36 0.00 4.00 .92 −.42
  Selection for EBP 2.30 1.00 0.00 4.00 −.28 −.36 .88 .15 .78
    13. Previously used EBP 2.12 1.14 0.00 4.00 −.15 −.57
    14. Formal education supporting EBP 2.28 1.09 0.00 4.00 −.20 −.53
    15. Value EBP 2.50 1.11 0.00 4.00 −.47 −.40
  Selection for Openness 2.83 .87 0.00 4.00 −.66 .36 .83 .18 .76
    16. Adaptable 2.98 .93 0.00 4.00 −.78 .51
    17. Flexible 3.00 .96 0.00 4.00 −.87 .62
    18. Open to new interventions 2.52 1.10 0.00 4.00 −.49 −.42
Open in a new tab

Measurement Invariance

A multi-group structural equation modeling approach was used to compare the factor loadings of the Implementation Climate Scale for participants in the child welfare sector versus the sample used in the CFA sample in Ehrhart, Aarons, and Farahnak’s (2014) research in the mental health sector (n = 630; see Ehrhart et al., 2014, for more detail on the sample). To test for weak factorial invariance across groups (Meredith, 1993), the chi-square from a model with all parameters allowed to be unequal across groups was compared to the chi-square from a model with only the loadings constrained to be equal across groups. No means or intercepts were estimated in these models. A summary of the model fit statistics is provided in Table 5. The model with all parameters freely estimated in the two groups fit the data somewhat well (CFI = .928; SRMR = .071; RMSEA = .073, 90% CI [.065–.080], Probability RMSEA ≤ .05 = .00) according to fit criteria suggested by Hu and Bentler (1999), although the overall chi-square was significant, χ2(252) = 594.928, p < .001. The metric invariance model with loadings constrained to be equal across groups generally fit the data worse than the model with parameters freely estimated (χ2(270) = 635.046, p < .001; CFI = .923; SRMR = .083; RMSEA = .072, 90% CI [.065–.080], Probability RMSEA ≤ .05 = .00). The Yuan-Bentler chi-square difference test resulted in a significant chi-square of Δχ2(18) = 40.06, p < .002.

Table 5.

Measurement invariance model comparison.

Fit Statistics Model Comparisons
CFI SRMR RMSEA χ 2 df Y-B Δχ2 Δdf ΔCFI ΔRMSEA
All parameters free
(configural
invariance)		 0.928 0.071 0.073 594.93 252 -- -- -- --
All loadings
constrained to equal
(metric invariance)		 0.923 0.083 0.072 635.05 270 40.06* 18 −0.005 −0.001
Open in a new tab
*

Notes: Y-B = Yuan-Bentler Chi-Square Difference Test; * p < .05

Although the findings based on the likelihood ratio chi-square test suggest that the measurement of the six-factor ICS differed between the mental health and child welfare samples, it is important to note that some researchers (Chen, 2007; Cheung & Rensvold, 2002; MacCallum, Browne, & Cai, 2005; Roesch, Norman, Merz, Sallis, & Patrick, 2013) have advocated for supplementing the chi-square difference test with information on the changes in the fit indices. For instance, the more stringent cutoffs for invariance recommended by Chen (2007) were that ΔCFI differences less than .005 and ΔRMSEA values less than .010 specify no meaningful difference between nested models. As shown in Table 5, the change in the CFI for the constrained and unconstrained models in this analysis was quite small (ΔCFI = .005) and the RMSEA value actually improved very slightly (ΔRMSEA = .001) because RMSEA is positively influenced when fewer parameters are estimated. Thus, we concluded that the metric invariance model holds and provides support for partial measurement invariance (weak factorial invariance) across sectors.

Discussion

The purpose of this study was to further validate the implementation climate scale by testing its factor structure in a sample of child welfare providers. Confirmatory factor analysis revealed that the previously validated measurement model in the mental health setting (Ehrhart, Aarons, & Farahnak, 2014) also demonstrates good model fit in the child welfare setting. Specifically, items loaded on the appropriate factors with acceptable factor loadings, confirming that the ICS is psychometrically sound in measuring EBP implementation climate across multiple healthcare settings. This is an important contribution to the literature, as focusing on a specific climate towards implementation emphasizes the need for support for implementation within the organization in order to achieve important EBP outcomes. A psychometrically sound scale of EBP implementation climate can supplement the organization’s implementation efforts by identifying areas of relative strength and weakness in terms of support for implementation within the organization.

Though not the primary focus, additional analyses were conducted to establish measurement equivalence of the ICS across child welfare and mental health sectors. Although the model comparisons across sectors were significantly different, the change in model fit was very small and the unstandardized loadings were generally similar in magnitude. Thus we concluded that the ICS exhibited weak factorial invariance, thus providing initial supportive evidence for using the ICS for comparisons between the child welfare and mental health sectors. For the purpose of this study, we chose to only test for partial measurement invariance. This was because equal factor loadings have been described as the prerequisite for making meaningful comparisons across groups (Ployhart & Oswald, 2004), and more restrictive models are argued as not necessary for measurement equivalence (Bryne & Stewart, 2006). This is understandable, as there are many similarities between the two settings, and although the specific EBPs in practice in child welfare and mental health settings may differ, the process for fostering support for the use of EBPs and thus the creation of an implementation climate is likely very similar. Contextual factors relevant to EBP implementation, such as organizational culture, leadership, and evaluation (McCormack, Kitson, Harvey, Rycroft-Malone, Titchen, Seers, 2002), have been found to be critical to successful sustainment in public health settings, and these contextual considerations are applicable to a wide array of organizational environments. Therefore, organizational context may influence the facilitation of implementation climate in a similar fashion within both child welfare and mental health.

There are some limitations to this study that should be addressed. One subscale of the ICS, Rewards for EBP, had weaker correlations with the other subscales and exhibited some inconsistency in the aggregation statistics. Although the within-group agreement (as indicated by the awg(j) value) was acceptable, the ICC(1) value for this scale was surprisingly low −.02. ICC(1) values are affected by both within-group and between-group variance, such that higher levels are found when within-group variability is lower and between-group variance is higher. Because the awg(j) value indicates that within-group variability was relatively low, the low ICC(1) is likely due to limited between-group variance. Our follow-up analyses confirmed that this was likely the problem; the standard deviation of the mean scores across groups was lower for the Rewards for EBP than for the other subscales (SD = .50 versus .60–.62 for the other subscales). This is also likely the explanation for the lower correlations with the other subscales. The mean for the Rewards for EBP subscale is consistently lower than the other subscales (as it was in the Ehrhart et al., 2014, validation paper in mental health), likely because resources are limited for providing financial incentives. Thus, the results indicate that teams generally similarly low on this characteristic, and thus it is characteristic of child welfare teams in general (or at least those in this sample) to not provide rewards for EBPs. An exploratory analysis removing this scale revealed slightly stronger model fit without it (χ2(80) = 157.611, p < .001; CFI = .95; SRMR = .060; RMSEA = .067, 90% CI [.052–.083], Probability RMSEA ≤ .05 = .036). Nevertheless, in our research we have identified some organizations that have used such incentives for EBP implementation, so we view this as an important aspect of EBP implementation climate, particularly with regard to its practical use in organizations. Future research should explore the contribution of rewards for EBP to the measurement of climate for EBP implementation, as well as how leaders in child welfare organizations may better utilize financial incentives to encourage EBP implementation.

In terms of other limitations and future directions, the ICS was created to measure EBP implementation climate across healthcare contexts, yet this study only examined one additional context. Future research should further demonstrate the generalizability of the ICS by testing it in other healthcare settings that regularly implement EBPs, such as nursing or substance use disorder treatment. Although this study addresses a crucial void in the literature by developing a multi-faceted approach to measuring EBP implementation climate, more knowledge is needed on how to create and sustain this climate across healthcare settings. Future research should address this need by measuring predictors of EBP implementation climate, such as availability of resources, EBP knowledge and skill, and leader facilitation. For instance, past research has demonstrated leaders can cultivate support for general climates (Lewin, Lippitt, & White, 1939) and specific climates within the organization (Jung, Wu, & Chow, 2008; Mayer, Nishii, Schneider, & Goldstein, 2007). Specifically with regard to implementation, Aarons, Ehrhart, Farahnak, and Sklar (2014) have described how leaders can utilize Schein’s (2010) embedding mechanisms to bolster the implementation climate in their organizations, and Aarons, Ehrhart, Farahnak, and Hurlburt (2015) describe leadership training specifically targeting improvement in EBP implementation. Research has also shown that development of strategic partnerships, the use of multiple strategies, and facilitator characteristics are associated with successful EBP implementation (Dogherty, Harrison, Graham, Vandyk, & Keeping-Burke, 2013). Future research could analyze the effectiveness of different types of leadership, strategies, and individual characteristic variables that either help to develop, foster, or impede EBP implementation climate.

Finally, although research in non-health-related settings (e.g., Klein et al., 2001) and in medical settings at the individual level (Jacobs et al., 2015) provide support for effects of EBP implementation climate on implementation outcomes, more research is needed to further demonstrate the benefits of cultivating such an organizational climate in child welfare and other allied health settings. Research of this nature could include other climate-related measures (e.g., Glisson et al., 2008) as well as measures of other aspects of the organizational context (e.g., organizational readiness for change; Helfrich, Li, Sharp, & Sales, 2009; Shea, Jacobs, Esserman, Bruce, & Weiner, 2014) to show how these various issues build on each other and work together to understand implementation effectiveness. For instance, there is evidence that molar climates can form a basis for which strategic climates to build upon (Salanova, Agut, & Peiró, 2005; Schneider, White, & Paul, 1998; Wallace, Popp, & Mondore, 2006), whereas others have suggested that molar climate has a direct relationship with outcomes (Williams & Glisson, 2014). In a similar way, it has been suggested that culture is an antecedent to climate (Ostroff, Kinicki, & Muhammad, 2012; Williams & Glisson, 2014), but some have also suggested that the nature of the relationship depends on the outcomes of interest (Schneider, Ehrhart, & Macey, 2011). Such research could include the effects of the specificity of the focus of the measure. The ICS was originally developed to focus on the climate for evidence-based practice in general, but it was also suggested that it could be adapted to measure the climate for implementation of specific EBPs (Ehrhart, Aarons, & Farahnak, 2014), in line with recommendations by Weiner et al. (2011). Such research could also take into account different measures of implementation climate, such as the recently developed measure by Jacobs et al. (2014), which was developed with physicians in a medical setting. The Jacobs et al. (2014) measure is six items and focuses more on the overall implementation climate, versus the ICS’s focus on more specific policies, practices, and procedures in the organization. Future research could compare the overlap and distinctiveness, particularly with regard to predicting implementation outcomes.

Conclusion

There is a need for a deeper understanding of the role of the organizational context in successful implementation of EBP in child welfare settings. A climate focused on EBP implementation may be vital to successful EBP implementation, as past research has demonstrated the role strategic climates can have on making a significant impact on targeted outcomes (Ehrhart, Witt, Schneider, & Perry, 2011; Schneider et al., 2005; Zohar & Luria, 2005). The present study’s results found support for the six-factor ICS structure in the child welfare sample, demonstrating that the scale is generalizable across different behavioral health settings. This study sets the foundation for further research on implementation climate in child welfare by providing support for a psychometrically sound measure of implementation climate. The dimensions of the ICS can be used not only as an evaluative tool, but also as a valid framework to develop and promote implementation climate within the child welfare setting. Future research can utilize these results to further EBP implementation success in the child welfare setting by validating the ICS with child welfare outcomes, identifying crucial correlates and predictors of implementation climate, and examining its interaction with other constructs related to the organizational context in predicting implementation effectiveness.

Acknowledgements

The preparation of this paper was supported by National Institute of Mental Health grants R21MH098124 (PI: Ehrhart), R21MH082731 (PI: Aarons), R01MH072961 (PI: Aarons), P30MH074678 (PI: Landsverk), R25MH080916 (PI: Proctor), and by the Child and Adolescent Services Research Center (CASRC) and the Center for Organizational Research on Implementation and Leadership (CORIL). The authors thank the community-based organizations, clinicians, and supervisors that made this study possible, as well as the additional research assistants (Brian Albanese, Natalie Finn) who supported this project.

Footnotes

Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

Contributor Information

Elisa M. Torres, Email: eltorres@ucsd.edu.

Lisa A. Wright, Email: lisa.wright.sdsu@gmail.com.

Sandra Y. Martinez, Email: sandra.martinez@mail.sdsu.edu.

Gregory A. Aarons, Email: gaarons@ucsd.edu.

References

  1. Aarons GA, Ehrhart MG, Farahnak LR. The implementation leadership scale (ILS): Development of a brief measure of unit level implementation leadership. Implementation Science. 2014;9:45. doi: 10.1186/1748-5908-9-45. [DOI] [PMC free article] [PubMed] [Google Scholar]
  2. Aarons GA, Ehrhart MG, Farahnak LR, Hurlburt MS. Leadership and organizational change for implementation (LOCI): A randomized mixed method pilot study of a leadership and organization development intervention for evidence-based practice implementation. Implementation Science. 2015;10:11. doi: 10.1186/s13012-014-0192-y. [DOI] [PMC free article] [PubMed] [Google Scholar]
  3. Aarons GA, Ehrhart MG, Farahnak LR, Sklar M. Aligning leadership across systems and organizations to develop a strategic climate for evidence-based practice implementation. Annual Review of Public Health. 2014;35:255–274. doi: 10.1146/annurev-publhealth-032013-182447. [DOI] [PMC free article] [PubMed] [Google Scholar]
  4. Aarons GA, Fettes DL, Hurlburt MS, Palinkas LA, Gunderson L, Willging CE, Chaffin MJ. Collaboration, negotiation, and coalescence for interagency-collaborative teams to scale-up evidence-based practice. Journal of Clinical Child & Adolescent Psychology. 2014;43(6):915–928. doi: 10.1080/15374416.2013.876642. [DOI] [PMC free article] [PubMed] [Google Scholar]
  5. Aarons GA, Glisson C, Green PD, Hoagwood K, Kelleher KJ, Landsverk JA & Research Network on Youth Mental Health. The organizational social context of mental health services and clinician attitudes toward evidence-based practice: a United States national study. Implementation Science. 2012;7(1):56. doi: 10.1186/1748-5908-7-56. [DOI] [PMC free article] [PubMed] [Google Scholar]
  6. Aarons GA, Green AE, Palinkas LA, Self-Brown S, Whitaker DJ, Lutzker JR, Chaffin MJ. Dynamic adaptation process to implement an evidence-based child maltreatment intervention. Implementation Science. 2012;7(32):1–9. doi: 10.1186/1748-5908-7-32. [DOI] [PMC free article] [PubMed] [Google Scholar]
  7. Aarons GA, Hurlburt M, Horwitz SM. Advancing a conceptual model of evidence-based practice implementation in public service sectors. Administration and Policy in Mental Health and Mental Health Services Research. 2011;38(1):4–23. doi: 10.1007/s10488-010-0327-7. [DOI] [PMC free article] [PubMed] [Google Scholar]
  8. Aarons GA, Palinkas LA. Implementation of evidence-based practice in child welfare: Service provider perspectives. Administration and Policy in Mental Health & Mental Health Services Research. 2007;34:411–419. doi: 10.1007/s10488-007-0121-3. [DOI] [PubMed] [Google Scholar]
  9. Aarons GA, Sommerfeld DH. Leadership, innovation climate, and attitudes toward evidence-based practice during a statewide implementation. Journal of the American Academy of Child & Adolescent Psychiatry. 2012;51(4):423–431. doi: 10.1016/j.jaac.2012.01.018. [DOI] [PMC free article] [PubMed] [Google Scholar]
  10. Barbee AP, Christensen D, Antle B, Wandersman A, Cahn K. Successful adoption and implementation of a comprehensive casework practice model in a public child welfare agency: Application of the Getting to Outcomes (GTO) model. Children and Youth Services Review. 2011;33(5):622–633. [Google Scholar]
  11. Brown RD, Hauenstein NM. Interrater agreement reconsidered: An alternative to the rwg indices. Organizational Research Methods. 2005;8(2):165–184. [Google Scholar]
  12. Byrne BM, Stewart SM. The MACS approach to testing for multigroup invariance of a second-order structure: A walk through the process. Structural Equation Modeling. 2006;13:287–321. [Google Scholar]
  13. Chen FF. Sensitivity of goodness of fit indexes to lack of measurement invariance. Structural Equation Modeling. 2007;14:464–504. [Google Scholar]
  14. Cheung GW, Rensvold RB. Evaluating goodness-of-fit indexes for testing measurement invariance. Structural Equation Modeling. 2002;9:233–255. [Google Scholar]
  15. Dogherty EJ, Harrison MB, Graham ID, Vandyk AD, Keeping - Burke L. Turning knowledge into action at the point-of-care: The collective experience of nurses facilitating the implementation of evidence-based practice. Worldviews on Evidence-Based Nursing. 2013;10(3):129–139. doi: 10.1111/wvn.12009. [DOI] [PMC free article] [PubMed] [Google Scholar]
  16. Dolan M, Smith K, Casanueva C, Ringeisen H. NSCAW II Baseline Report: Caseworker Characteristics, Child Welfare Services, and Experiences of Children Placed in Out-of-Home Care. #OPRE Report #2011-27e. Washington, DC: Office of Planning, Research and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services; 2011. [Google Scholar]
  17. Ehrhart KH, Witt LA, Schneider B, Perry SJ. Service employees give as they get: Internal service as a moderator of the service climate-service outcomes link. Journal of Applied Psychology. 2011;96(2):423–431. doi: 10.1037/a0022071. [DOI] [PubMed] [Google Scholar]
  18. Ehrhart MG, Aarons GA, Farahnak LR. Assessing the organizational context for EBP implementation: The development and validity testing of the Implementation Climate Scale (ICS) Implementation Science. 2014;9:157. doi: 10.1186/s13012-014-0157-1. [DOI] [PMC free article] [PubMed] [Google Scholar]
  19. Ehrhart MG, Schneider B, Macey WH. Organizational climate and culture: An introduction to theory, research, and practice. New York: Routledge; 2014. [Google Scholar]
  20. Fogarty International Center. Frequently Asked Questions about Implementation Science. 2013 May; From http://www.fic.nih.gov/News/Events/implementation-science/Pages/faqs.aspx.
  21. Garland AF, Hurlburt MS, Hawley KM. Examining psychotherapy processes in a services research context. Clinical Psychology: Science And Practice. 2006;13(1):30–46. [Google Scholar]
  22. Guo Q, Li F, Chen X, Wang W, Meng Q. Performance of fit indices in different conditions and selection of cut-off values. Acta Psychologica Sinica. 2008;40(1):109–118. [Google Scholar]
  23. Glisson C, Green P. Organizational climate, services, and outcomes in child welfare systems. Child Abuse & Neglect. 2011;35(8):582–591. doi: 10.1016/j.chiabu.2011.04.009. [DOI] [PubMed] [Google Scholar]
  24. Glisson C, Hemmelgarn A. The effects of organizational climate and interorganizational coordination on the quality and outcomes of children’s service systems. Child Abuse & Neglect. 1998;22(5):401–421. doi: 10.1016/s0145-2134(98)00005-2. [DOI] [PubMed] [Google Scholar]
  25. Glisson C, Green P, Williams NJ. Assessing the Organizational Social Context (OSC) of child welfare systems: implications for research and practice. Child abuse & neglect. 2012;36(9):621–632. doi: 10.1016/j.chiabu.2012.06.002. [DOI] [PubMed] [Google Scholar]
  26. Glisson C, James LR. The cross-level effects of culture and climate in human service teams. Journal of Organizational Behavior. 2002;23(6):767–794. [Google Scholar]
  27. Glisson C, Landsverk J, Schoenwald S, Kelleher K, Hoagwood KE, Mayberg S, Green P. Assessing the organizational social context (OSC) of mental health services: Implications for research and practice. Administration and Policy in Mental Health and Mental Health Services Research. 2008;35(1–2):98–113. doi: 10.1007/s10488-007-0148-5. [DOI] [PubMed] [Google Scholar]
  28. Helfrich C, Li Y, Sharp N, Sales A. Organizational readiness to change assessment (ORCA): Development of an instrument based on the promoting action on research in health services (PARIHS) framework. Implementation Science. 2009;4(1):38. doi: 10.1186/1748-5908-4-38. [DOI] [PMC free article] [PubMed] [Google Scholar]
  29. Holmes L, Landsverk J, Ward H, Rolls-Reutz J, Saldana L, Wulczyn F, Chamberlain P. Cost calculator methods for estimating casework time in child welfare services: A promising approach for use in implementation of evidence-based practices and other service innovations. Children and Youth Services Review. 2014;39:169–176. doi: 10.1016/j.childyouth.2013.10.003. [DOI] [PMC free article] [PubMed] [Google Scholar]
  30. Horwitz SM, Chamberlain P, Landsverk J, Mullican C. Improving the mental health of children in child welfare through the implementation of evidence-based parenting interventions. Administration and Policy in Mental Health and Mental Health Services Research. 2010;37(1–2):27–39. doi: 10.1007/s10488-010-0274-3. [DOI] [PubMed] [Google Scholar]
  31. Hu LT, Bentler PM. Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural equation modeling: A Multidisciplinary Journal. 1999;6(1):1–55. [Google Scholar]
  32. Jacobs SR, Weiner BJ, Bunger AC. Context matters: Measuring implementation climate among individuals and groups. Implementation Science. 2014;9:46. doi: 10.1186/1748-5908-9-46. [DOI] [PMC free article] [PubMed] [Google Scholar]
  33. Jacobs SR, Weiner BJ, Reeve BB, Hofmann DA, Christian M, Weinberger M. Determining the predictors of innovation implementation in healthcare: a quantitative analysis of implementation effectiveness. BMC health services research. 2015;15(1):6. doi: 10.1186/s12913-014-0657-3. [DOI] [PMC free article] [PubMed] [Google Scholar]
  34. Jung D, Wu A, Chow CW. Towards understanding the direct and indirect effects of CEOs' transformational leadership on firm innovation. The Leadership Quarterly. 2008;19(5):582–594. [Google Scholar]
  35. Kaye S, DePanfilis D, Bright CL, Fisher C. Applying implementation drivers to child welfare systems change: Examples from the field. Journal of Public Child Welfare. 2012;6(4):512–530. [Google Scholar]
  36. Kessler M, Gira E, Poertner J. Moving best practice to evidence-based practice in child welfare. Families in Society-the Journal of Contemporary Social Services. 2005;86(2):244–250. [Google Scholar]
  37. Klein KJ, Conn AB, Sorra JS. Implementing computerized technology: An organizational analysis. Journal of Applied Psychology. 2001;86:811–824. doi: 10.1037/0021-9010.86.5.811. [DOI] [PubMed] [Google Scholar]
  38. Kolko DJ, Baumann BL, Herschell AD, Hart JA, Holden EA, Wisniewski SR. Implementation of AF-CBT by community practitioners serving child welfare and mental health: A randomized trial. Child maltreatment. 2012 doi: 10.1177/1077559511427346. 1077559511427346. [DOI] [PubMed] [Google Scholar]
  39. Kuenzi M, Schminke M. Assembling fragments into a lens: A review, critique, and proposed research agenda for the organizational work climate literature. Journal of Management. 2009;35:634–717. [Google Scholar]
  40. Lewin K, Lippitt R, White RK. Patterns of aggressive behavior in experimentally created 'social climates'. The Journal of Social Psychology. 1939;10:271–299. [Google Scholar]
  41. Little TD. Means and covariance structures (MACS) analyses of crosscultural data: Practical and theoretical issues. Multivariate Behavioral Research. 1997;32:53–76. doi: 10.1207/s15327906mbr3201_3. [DOI] [PubMed] [Google Scholar]
  42. Lobb R, Colditz G. Implementation science and its application to population health. Annual Review of Public Health. 2013;34(1):235–251. doi: 10.1146/annurev-publhealth-031912-114444. [DOI] [PMC free article] [PubMed] [Google Scholar]
  43. MacCallum RC, Browne MW, Cai L. Tested differences between nested covariance structure models: Power analyses and null hypotheses. Psychological Methods. 2005;2011:19–35. doi: 10.1037/1082-989X.11.1.19. [DOI] [PubMed] [Google Scholar]
  44. Mayer D, Nishii L, Schneider B, Goldstein H. The precursors and products of justice climates: Group leader antecedents and employee attitudinal consequences. Personnel Psychology. 2007;60(4):929–963. [Google Scholar]
  45. McCormack B, Kitson A, Harvey G, Rycroft-Malone J, Titchen A, Seers K. Getting evidence into practice: The meaning of “context”. Journal of Advanced Nursing. 2002;38:94–104. doi: 10.1046/j.1365-2648.2002.02150.x. [DOI] [PubMed] [Google Scholar]
  46. Meredith W. Measurement invariance, factor analysis, and factorial invariance. Psychometrika. 1993;58:525–543. [Google Scholar]
  47. Mildon R, Shlonsky A. Bridge over troubled water: Using implementation science to facilitate effective services in child welfare. Child abuse & neglect. 2011;35(9):753–756. doi: 10.1016/j.chiabu.2011.07.001. [DOI] [PubMed] [Google Scholar]
  48. Mitchell P. Evidence-based practice in real-world services for young people with complex needs: New opportunities suggested by recent implementation science. Children and Youth Services Review. 2011;33(2):207–216. [Google Scholar]
  49. Muthén LK, Muthén BO. Mplus user’s guide. 7th edition. Los Angeles, CA: Muthén & Muthén; 1998–2012. [Google Scholar]
  50. Ostroff C, Kinicki AJ, Muhammad RS. Organizational culture and climate. In: Schmitt NW, Highhouse S, editors. Handbook of psychology, Vol. 12: Industrial and organizational psychology. 2nd Ed. Hoboken, NJ: Wiley; 2012. pp. 643–676. [Google Scholar]
  51. Pipkin S, Sterrett EM, Antle B, Christensen DN. Washington State’s adoption of a child welfare practice model: An illustration of the getting to outcomes implementation framework. Children and Youth Services Review. 2013;35(12):1923–1932. [Google Scholar]
  52. Pecora PJ, Whittaker JK, Maluccio AN, Barth RP. The child welfare challenge: Policy, practice, and research. AldineTransaction. 2012 [Google Scholar]
  53. Ployhart RE, Oswald FL. Applications of mean and covariance structure analysis: Integrating correlational and experimental approaches. Organizational Research Methods. 2004;7:27–65. [Google Scholar]
  54. Proctor LK, Landsverk J, Aarons GA, Chambers D, Glisson C, Mittman B. Implementation research in mental health services: an emerging science with conceptual, methodological, and training challenges. Administrative Policy in Mental Health and Mental Health Services. 2009;36:24–34. doi: 10.1007/s10488-008-0197-4. [DOI] [PMC free article] [PubMed] [Google Scholar]
  55. Roesch SC, Norman GJ, Merz EL, Sallis JF, Patrick K. Longitudinal measurement invariance of psychosocial measures in physical activity research: An application to adolescent data. Journal of Applied Social Psychology. 2013;43(4):721–729. doi: 10.1111/jasp.12000. [DOI] [PMC free article] [PubMed] [Google Scholar]
  56. Salanova M, Agut S, Peiró JM. Linking organizational resources and work engagement to employee performance and customer loyalty: The mediation of service climate. Journal of Applied Psychology. 2005;90:1217–1227. doi: 10.1037/0021-9010.90.6.1217. [DOI] [PubMed] [Google Scholar]
  57. Schein EH. Organizational culture and leadership. 4th ed. San Francisco: Jossey-Bass; 2010. [Google Scholar]
  58. Schneider B, Ehrhart MG, Macey WH. Organizational climate research: Achievements and the road ahead. In: Ashkanasy NM, Wilderom CPM, Peterson MF, editors. The handbook of organizational culture and climate. 2nd ed. Thousand Oaks, CA: Sage; 2011. pp. 29–49. [Google Scholar]
  59. Schneider B, Ehrhart MG, Mayer DM, Saltz JL, Niles-Jolly K. Understanding organization-customer links in service settings. Academy of Management Journal. 2005;48(6):1017–1032. [Google Scholar]
  60. Schneider B, White SS, Paul MC. Linking service climate and customer perceptions of service quality: Test of a causal model. Journal of Applied Psychology. 1998;83:150–163. doi: 10.1037/0021-9010.83.2.150. [DOI] [PubMed] [Google Scholar]
  61. Shea CM, Jacobs SR, Esserman DA, Bruce K, Weiner BJ. Organizational readiness for implementing change: A psychometric assessment of a new measure. Implementation Science. 2014;9(7):1–15. doi: 10.1186/1748-5908-9-7. [DOI] [PMC free article] [PubMed] [Google Scholar]
  62. Stirman S, Crits-Christoph P, DeRubeis R. Achieving successful dissemination of empirically supported psychotherapies: A synthesis of dissemination theory. Clinical Psychology: Science and Practice. 2004;11(4):343–359. [Google Scholar]
  63. Wallace JC, Popp E, Mondore S. Safety climate as a mediator between foundation climates and occupational accidents: A group-level investigation. Journal of Applied Psychology. 2006;91:681–688. doi: 10.1037/0021-9010.91.3.681. [DOI] [PubMed] [Google Scholar]
  64. Weiner B, Belden C, Bergmire D, Johnston M. The meaning and measurement of implementation climate. Implementation Science. 2011;6(1):78. doi: 10.1186/1748-5908-6-78. [DOI] [PMC free article] [PubMed] [Google Scholar]
  65. Williams NJ, Glisson C. Testing a theory of organizational culture, climate and youth outcomes in child welfare systems: A United States national study. Child Abuse & Neglect. 2014;38(4):757–767. doi: 10.1016/j.chiabu.2013.09.003. [DOI] [PMC free article] [PubMed] [Google Scholar]
  66. Zohar D, Luria G. A multilevel model of safety climate: Cross-level relationships between organization and group-level climates. Journal of Applied Psychology. 2005;90(4):616–628. doi: 10.1037/0021-9010.90.4.616. [DOI] [PubMed] [Google Scholar]

RESOURCES