Implementing effective policy in a national mental health re-engagement program for Veterans

Abstract

Policy is a powerful motivator of clinical change, but implementation success can depend on organizational characteristics. This paper used validated measures of organizational resources, culture and climate to predict uptake of a nationwide VA policy aimed at implementing Re-Engage, a brief care management program that re-establishes contact with Veterans with serious mental illness lost to care. Patient care databases were used to identify 2,738 Veterans lost to care. Local Recovery Coordinators (LRCs) were to update disposition for 2,738 Veterans at 158 VA facilities and, as appropriate, facilitate a return to care. Multivariable regression assessed organizational culture and climate as predictors of early policy compliance (via LRC presence) and uptake at six months. Higher composite climate and culture scores were associated with higher odds of having a designated LRC, but were not predictive of higher uptake. Sites with LRCs had significantly higher rates of updated documentation than sites without LRCs.

INTRODUCTION

Changes in health policy can be a strong motivator for changes in mental health care practice, yet even unified policy begets uneven implementation and interpretation. Prior scholars have pointed to the ‘implementation conundrum’ in health policy, or the failure of government to ensure implementation of the policies it passes (Siler-Wells, 1987). Much of this failure, however, can be attributed to organizational characteristics that facilitate or hinder successful implementation, including leadership priorities (Aarons et al, 2014; Aarons et al, 2011), staff capacity and power to facilitate change (Helfrich et al, 2009; Weiner, 2009), size and maturity (Damschroder et al, 2009), culture or predominant values (Cameron et al, 2005; Denison et al, 1991; Shortell et al, 2001; Speroff et al, 2010), and climate (Benzer et al, 2011; Ehrhart et al, 2014; Glisson et al, 1998; Lawler et al, 1974). As yet, however, few studies have examined the impact of organizational factors on the success of policy-mandated, planned implementation success in real-world settings.

In early 2012, the Veteran’s Health Administration (VA) issued a policy requiring all VA sites to implement a new brief care management program for Veterans with serious mental illness, the ‘Re-Engage’ program. This paper examines the effects of organizational characteristics, including culture, climate and staffing, on compliance with this policy, both initially and further downstream. Specifically, we link data on initial policy compliance (via presence of a designated staff member responsible for program implementation) and six-month implementation outcomes with site-level measures of organizational culture and climate to examine the impact of organizational characteristics on processes of policy compliance and program implementation.

A vast literature exists connecting formal and structural organizational characteristics, including size, resources and leadership, to implementation success. In addition, organizational culture and climate have been identified as key predictors of evidence-based practice dissemination and implementation across health care organizations (Damschroder et al, 2009; Piña et al, 2015), distinct from these other characteristics. As organizational culture and climate are characterized as generally stable (Denison et al, 1991; Quinn et al, 1991) and their effects as subconscious (Helfrich et al, 2007), research that better elucidates which types of culture or climate are important, and through which mechanisms, may help to ensure implementation success across diverse organizational settings.

Organizational culture

Organizational culture represents the shared norms, values, and expectations that govern organizational behavior (Ostroff et al, 2003; Scott et al, 2003). Organizational culture is stable and difficult to change (Denison et al, 1991; Quinn et al, 1991); it imbues workplaces with their priorities of practice, affecting individual work processes in subtle ways (Helfrich et al, 2007). Measures of organizational culture have been linked to differences in organizational practice (Krein et al, 2002; Strasser et al, 2002), provider effectiveness (Gifford et al, 2002), patient care quality (Davies et al, 2007; Rondeau et al, 1998), improved patient health (Shortell, 2004; Shortell et al, 2000), attitudes towards evidence-based practice (Callen et al, 2007) and implementation of quality care (Shortell et al, 1995).

In context of implementing the Re-Engage directive, we focused on culture emphasizing innovation and change, as compared to culture emphasizing stability and aversion to change. Prior work has found organizations valuing flexibility, innovation, and teamwork more receptive to quality improvement (Cameron et al, 2005; Davies et al, 2007; Hartmann et al, 2009; Meterko et al, 2004; Shortell, 2004; Shortell et al, 2001; Speroff et al, 2010; Zazzali et al, 2007) and implementation of clinical innovation (Berlowitz et al, 2003; Carman et al, 1996; Shortell et al, 2001). Alternatively, organizations valuing stability and permanence have been found less receptive of these same endeavors (Berlowitz et al, 2003; Carroll et al, 2002; Hartmann et al, 2009; Shortell et al, 1995; Speroff et al, 2010), though also better at sustaining change once it occurs (Cole et al, 2000).

Measures of entrepreneurial culture and hierarchical culture are defined in Quinn and Rohrbaugh’s Cultural Values Framework, which defines cultural archetypes through two dimensions of competing values: centralized vs. de-centralized and internal focus vs. external focus (Quinn et al, 1981). Entrepreneurial cultures value flexibility and creativity over control, and adaptation to external environments and adoption of new practices over internal cohesion (Helfrich et al, 2007; Kalliath et al, 1999). Health services researchers have found higher levels of entrepreneurial culture predictive of positive safety climates (Hartmann et al, 2009), clinicians accepting responsibilities beyond formal job description (Williams et al, 2007), and staff satisfaction (Lin et al, 2012).

Hierarchical cultures, alternatively, respect formal hierarchy and authority, and prioritize predictability, stability, and adherence to rules (Helfrich et al, 2007; Kalliath et al, 1999). In health services research, hierarchical culture has been associated with poorer safety climates (Hartmann et al, 2009; Singer et al, 2009), staff turnover (San Park et al, 2009), lower patient satisfaction (Meterko et al, 2004), and poorer implementation of quality improvement (Ferlie et al, 2001), but also greater patient commitment and trust (Williams et al, 2007).

Prior work on organizational culture in health care settings, including the VA, has shown positive correlations between entrepreneurial and hierarchical measures of culture (Helfrich et al, 2007; Kalliath et al, 1999). VA scholars noted this may indicate a lack of distinction between extrinsic and intrinsic motivation amongst VA employees (Helfrich et al, 2007). More broadly, a positive correlation between entrepreneurial and hierarchical culture may highlight the importance of endorsing multiple types of culture, rather than the predominance of one, for optimal patient care and quality improvement (Bradley et al, 2006; Denison et al, 1991; Helfrich et al, 2007; Shortell, 2004). Given these findings, we also considered composite culture, or summative entrepreneurial and hierarchical culture subscale scores, as predictive of Re-Engage compliance and implementation.

Organizational climate

Organizational climate represents employees’ shared perceptions of an organization’s policies, practices, and procedures. Organizational climate links organizational culture with employee behaviors and attitudes (James et al, 2008; Jones et al, 1979). Measures of climate will capture, for example, perceptions of communication norms with colleagues and supervisors, and organizational priorities (e.g., change or stability) as viewed by employees.

Like organizational culture, organizational climate is a collective and multifaceted construct, reflecting shared perceptions of the environment; unlike culture, climate is generally considered mutable (James et al, 1981; Jones et al, 1979). In health care settings, organizational climate has been linked to organizational innovation (Anderson et al, 1998; Nystrom et al, 2002; Wilson et al, 1999), program implementation (Allen et al, 2007), employee creativity (Choi et al, 2009), and effective management of preventive care and care of chronic conditions (Benzer et al, 2011).

Benzer et al. defined specific measures of task and relational climate for use in predicting quality of chronic care (Benzer et al, 2010; Benzer et al, 2013; Benzer et al, 2011). Task climate is defined by employee perception of a managerial focus on improvement and achievement. Relational climate, alternatively, captures employees perceptions of a managerial focus on mutual support and respect (Benzer et al, 2011). Relational climate has been associated with higher quality provision of chronic care (Benzer et al, 2013; Benzer et al, 2011) and patient ratings of quality of care (Mohr et al, 2013). As with culture, we also evaluated composite task and relational climate as predictive of implementation success.

The Re-Engage Policy Directive and Program Implementation

The Re-Engage program, described elsewhere (Goodrich et al, 2012; Kilbourne et al, 2014a; Kilbourne et al, 2013; Kilbourne et al, 2014c; Smith et al, 2011; Veterans Affairs; Veterans Affairs Office of the Medical Inspector, 2010) is a brief care management program developed by the VA Office of the Medical Inspector (OMI) to “re-engage” Veterans with SMI (defined as bipolar disorder, schizophrenia, or schizophrenia-related disorder) lost to follow-up care for more than one year with VA healthcare services (Veterans Health Administration, 2012). Re-Engage specifically targeted this group as, when SMI is left untreated due to gaps in continuity of care with health providers, patients with SMI experience disproportionate rates of early mortality from conditions like cancer and cardiovascular disease (Crump et al, 2013; Kilbourne et al, 2009; Saha et al, 2007). Barriers to care access and gaps in care continuity exacerbate vulnerability to mortality or morbidity from preventable health conditions (Copeland et al, 2006; McCarthy et al, 2007). The Re-Engage policy sought to remedy these disparities (Kreyenbuhl et al, 2009). An initial quality improvement demonstration project found significantly lower mortality for Veterans with SMI who returned to care compared with Veterans who did not (0.5 percent versus 3.9 percent), after accounting for clinical and demographic factors (Davis et al, 2012; Veterans Affairs Office of the Medical Inspector, 2010).

On January 10, 2012, VHA Directive 2012-002 was issued requiring all VA sites to “re-engage in treatment Veterans with serious mental illness that have been lost to follow-up care” (Kreyenbuhl et al, 2009; Veterans Health Administration, 2012). Veterans with SMI were designated as “lost to care” if they had ever received VA inpatient or outpatient care but had not received treatment (no outpatient treatment or an inpatient stay of two days or fewer) in the past 12 months. All VA facilities in the United States and Puerto Rico with at least one Veteran with SMI lost to care were required to implement the Re-Engage program protocol. 158 sites were eligible for Re-Engage, including 139 medical centers and 19 large community-based outpatient clinics.

Local Recovery Coordinators (LRCs) oversaw implementation of the Re-Engage directive. The LRCs position was established in December 2006, with LRCs responsible for overseeing the transformation of VHA’s mental health services to a recovery orientation, and ensuring that Veterans with SMI receive the services to which they are entitled. However, while the requirement of the LRC position pre-dated the Re-Engage directive, after establishment of the LRC role, some LRCs were not able to allocate as much time as expected to recovery-oriented tasks due to competing demands. Further, as of early 2012, not all VAMCs had an LRC in place. The Re-Engage Directive explicitly sought to reinforce the LRC’s role in providing recovery-oriented care, requiring VA Medical Directors to ensure that “re-engagement services take precedence over or are provided in lieu of other clinical assignments” for LRCs (Administration, 2012).

Implementation of the Re-Engage policy began on March 1, 2012. LRCs at each VA facility were provided with a list of Veterans with SMI lost to care and in need of outreach (see Directive (Administration, 2012) and (Kilbourne et al, 2014a) for detailed eligibility criteria). Veterans were assigned to facility lists based on location of last visit prior to dropout. LRCs were expected to document each listed Veteran’s current disposition in a database maintained by the VA Serious Mental Illness Treatment Resource and Evaluation Center (SMITREC; a program evaluation center within the VA Office of Mental Health Operations) and, if appropriate, attempt to contact the Veteran (Administration, 2012). For those Veterans successfully contacted, LRCs assessed clinical need and facilitated a return to VA care by scheduling an appointment (Kilbourne et al, 2014b). As part of a broader implementation trial all LRCs received training and technical support for implementing Re-Engage based on the Replicating Effective Programs model (Kilbourne et al, 2007; Neumann et al, 2000) throughout the first six months.

Paper aims and hypotheses

This paper has three aims. Aim 1 is to validate two measures of organizational culture (entrepreneurial and hierarchical) and two measures of organizational climate (task and relational) using data from the 2012 VA All Employee Survey (AES). Aims 2 and 3 examine whether the relationship between these measures of culture and climate and successful compliance with the Re-Engage policy. In Aim 2, we examine measures of organizational culture and climate as predictive of compliance with VA directives establishing an LRC, measured one week prior to Re-Engage rollout. Aim 3 evaluates the relationship between organizational culture, organizational climate, and presence of an LRC and success in implementing the Re-Engage policy after six months.

Implementation was measured with two outcomes: (1) percentage of eligible Veterans whose documentation had been updated to reflect current disposition (primary implementation outcome); and, for sites who had updated documentation for at least one listed Veteran, (2) the percentage of listed Veterans with a completed contact (secondary implementation outcome). Hypotheses linking each of these outcomes to the culture, climate and LRC variables are listed in Table 1.

Table 1.

Hypothesized direction of relationship between organizational characteristics and study outcomes

	POLICY COMPLIANCE	IMPLEMENTATION
	Outcome 1: Designated LRC?	Outcome 2: Site % updated disposition	Outcome 3: Site % completed contacts
CULTURE
Entrepreneurial	+	+	+
Hierarchical	−	−	−
Composite (Entrepreneurial+ Hierarchical)	+	+	+
CLIMATE
Task	+	+	+
Relational	+	+	+
Composite (Task+Relational)	+	+	+
LRC PRESENCE	N/A	+	+

METHODS

Setting

This study included 158 VA facilities in the United States and Puerto Rico required to implement VA Policy Directive 2012-002. Figure 1 shows the full study and data collection timeline. Compliance with VA policy, operationalized by presence of an LRC, was measures in late February 2012, just prior to Re-Engage program rollout. To allow evaluation of the effects of organizational characteristics on policy compliance and implementation, site-level data on LRC presence, number of listed Veterans, and patient demographics was combined with six-month implementation outcomes measured on September 1, 2012, and site-level measures of culture and climate from the 2012 VA All Employee Survey.

Figure 1.

Study timeline, including directive passage, program roll-out, and data collection

Data on culture and climate: The VA All Employee Survey (AES)

Implementation-specific measures of culture and/or climate (Aarons et al, 2014; Ehrhart et al, 2014) were not collected as part of the Re-Engage program. However, data on culture and climate for all VA sites was available through the VA All Employee Survey (AES). The AES is an annual survey of all VA employees at all locations that queries workplace satisfaction, culture, climate, turnover, supervisory behaviors, and general workplace perceptions. Respondents are asked to reflect on the past six months when answering the survey. In 2012 the survey was fielded from April 23 and May 14, 2012, yielding 173,413 responses, with a 63.4% response rate. As study interest was in site-level measures of organizational culture and climate, we used aggregated site-level data available from the VA Employee Survey Portal, part of the VA Service Support Center. These responses represent the 139 VA hospitals and 19 community-based outreach clinics required to implement Re-Engage.

The AES includes a battery of questions on ‘Culture’ that corresponds to Competing Values Framework measures. The 2012 AES included four items for each of hierarchical and entrepreneurial archetypes, querying all relevant dimensions. The AES also included an ‘Organizational Assessment Inventory’ with three items related to each of task climate and relational climate. All items are listed in Table 2, and were measured on a five-point Likert scale ranging from Strongly Disagree (1) to Strongly Agree (5).

Table 2.

Items and factor loadings for culture and climate measures from confirmatory factor analysis

	STANDARDIZED FACTOR LOADINGS IN BEST-FITTING CFA
	Entrepreneurial culture	Hierarchical culture	Task climate	Relational climate
ENTREPRENEURIAL CULTURE
1. My facility is a very dynamic and entrepreneurial place. People are willing to stick their necks out and take risks.	0.950	--
2. Managers in my facility are risk- takers. They encourage employees to take risks and be innovative.	0.923	--
3. The glue that holds my facility together is commitment to innovation and development. There is an emphasis on being first.	0.966	--
4. My facility emphasizes growth and acquiring new resources. Readiness to meet new challenges is important.	0.954	--
HIERARCHICAL CULTURE
5. My facility is a very formalized and structured place. Bureaucratic procedures generally govern what people do.	--	0.849
6. Managers in my facility are rule- enforcers. They expect employees to follow established rules, policies and procedures.	--	0.845
7. The glue that holds my facility together is formal rules and policies. People feel that following the rules is important.	0.379	0.937
8. My facility emphasizes permanence and stability. Keeping things the same is important.	--	<Dropped>
TASK CLIMATE
9. New practices and ways of doing business are encouraged in my work.			0.952	--
10. Managers set challenging and yet attainable performance goals for my work group.			0.931	--
11. Employees in my work group are involved in improving the quality of products, services, and work processes.			0.984	--
RELATIONAL CLIMATE
12. Disputes or conflicts are resolved fairly in my work group.			--	0.951
13. A spirit of cooperation and teamwork exists in my work group.			--	0.953
14. Differences among individuals are respected and valued in my work group.			--	0.963
FACTOR CORRELATION	0.10		0.92
FIT STATISTICS
CFI:	0.90		0.97
NNFI:	0.89		0.95
SRMR:	0.076		0.016

Note: CFI=Comparative Fit Index; NNFI=Bentler-Bonnett Non-normed Index; SRMR=standardized root-square mean residual

Local Recovery Coordinator Presence

As the Re-Engage policy specifically protected time for coordinated implementation, a designated LRC signaled an initial commitment to Re-Engage implementation, as well as either continued compliance with the 2006 directive that established LRCs, or with the Re-Engage directive. Presence of a designated LRC was measured one week prior to program rollout and was used as both an outcome for Aim 2 (to assess the role of organizational culture and climate in predicting VA policy compliance) and a predictor of successful implementation outcomes in Aim 3.

Implementation outcomes

At six months, policy compliance was again measured via two outcomes related to the implementation of Re-Engage by LRCs. Prior to attempting Veteran outreach, LRCs were first responsible for updating documentation on each Veterans’ current disposition, based on medical record review. The first outcome was the percentage of listed Veterans whose disposition had been documented by LRCs at six months. For sites with updated disposition for at least one Veteran, the second outcome was the percentage of Veterans with whom the LRC had a completed contact at six months. Both outcomes were collected by SMITREC via LRC completion of a Record of Re-Engagement Activities for each listed Veteran. As a standard part of the SMI Re-Engage program, LRCs are required to use this record to document all re-engagement efforts (including updated disposition) and their outcomes for each Veteran. The data is used for the monitoring and evaluation of the Re-Engage initiative and was obtained for the present study.

Analysis strategy for primary and secondary aims

The first aim of this paper was to validate measures of culture and climate for the VA sites included in the Re-Engage study. Cronbach’s alpha was first used to test for internal consistency. Confirmatory factor analysis (CFA) was then used to confirm the two-factor structure specified by the Competing Values Framework for measures of entrepreneurial and hierarchical culture; and Benzer et al. (Benzer et al, 2011) for task and relational climate. SAS 9.4 CALIS procedure with full maximum likelihood was used for analyses.

Aims 2 and 3 assess the relationship between the culture and climate measures developed in Aim 1 and both initial compliance and compliance at six months with the Re-Engage directive. Aim 2 uses binary logit models to examine the relationship between culture and climate and early compliance with the Re-Engage directive, measured as presence of a designated LRC directly prior to rollout. Predicted probabilities are also computed, to better illustrate the effect of certain variables on the probability of having a designated LRC. Aim 3 uses linear regression to examine the relationship between culture, climate and LRC presence and compliance at six months, measured by site-level percentage of listed Veterans with (1) updated disposition; and (2) completed contact. Control variables for facility size (in thousands of patients) and geographic region (categorized as Midwest, South, Northeast and West) were included for all models. For models assessing implementation outcomes, a further control for total number of Veterans listed for follow-up was also included to account for LRC burden. As successful contact completion is a function of both successful implementation and patient population characteristics, the model for completed contacts also controlled for the percentage of patients at each facility that were homeless, had a diagnosis of schizophrenia, identified as black/African-American, and had a service-connected disability. All patient characteristics were derived from administrative data gathered as part of the patients’ prior VA visits.

RESULTS

Measures of organizational culture and climate for the VA setting

The four entrepreneurial culture subscale items showed high internal consistency (Cronbach’s alpha=0.97); four hierarchical subscale items showed lower internal consistency (Cronbach’s alpha=0.75). Modification indices suggested removing one item from the hierarchical subscale (#8 in Table 2), after which Cronbach’s alpha improved to 0.91.

CFA was then run on the four items from the entrepreneurial culture subscale and three remaining items from the hierarchical subscale. Two factors—entrepreneurial culture and hierarchical culture—were specified, with loadings only for pertinent subscale measures. Fit statistics for this model fell short of established criteria for well-fitting models. Modification indices suggested loading hierarchical subscale item #7 on both factors. A second CFA specifying two factors and cross-loading item #7 met standards of ‘adequate’ fit for structural validity (Lewis et al, 2015). Standardized factor loadings for the seven-item, two-factor CFA for culture can be found in Table 2. This solution found a small positive correlation between factors of 0.10.

Cronbach’s alpha indicated high internal cohesion for both task (0.97) and relational climate (0.97) subscales. A two-factor CFA, with each factor loading onto relevant subscale items, fit the data well. Standardized factor loadings can be found in Table 2. As in prior work, the task and relational climate factors were highly correlated, at 0.92 (Benzer et al, 2011; James et al, 1989).

Factor loadings shown in Table 2 were used to construct factor scores for each culture and climate measure, with high scores indicating higher levels of the particular culture or climate. Factor scores were centered, with mean of 0. As expected, high collinearity was found for both relational and task climate measures (r=0.94), and entrepreneurial and hierarchical culture measures (r=0.69), further validating consideration of composite measures of organizational culture and climate. To ease interpretation, composite measures were standardized with mean of 0 and standard deviation of 1. Descriptive statistics for these are shown in Table 3. Composite culture and climate scores also showed a moderate positive correlation (r=0.60).

Table 3.

Characteristics of study sites (N=158)

	% (N) or Mean	SD	Minimum Value	Maximum Value
CULTURE & CLIMATE
Entrepreneurial culture	0	8.33	−21.69	19.30
Hierarchical culture	0	10.52	−28.36	33.66
Composite culture (standardized)	0	1	−2.72	2.76
Task climate	0	7.74	−17.49	19.80
Relational climate	0	7.98	−16.35	24.94
Composite climate (standardized)	0	1	−2.10	2.80
OUTCOMES
Has Local Recovery Coordinator (LRC)?
Yes	85% (134)
No	15% (24)
% updated documentation	54%	41%	0%	100%
% completed contact (N=116)	33%	26%	0%	100%
PREDICTORS
Region
Northeast	26% (41)
Midwest	24% (38)
South	28% (44)
West	22% (35)
Total number of patients  at site (FY 2012)	40,858	21,346	8,378	101,801
Number of patients  diagnosed with serious  mental illness lost to  care on site list	17.3	6.36	3	44
% of patient population  reported as:
Black	16%	--	0%	73%
Homeless	16%	--	0%	60%
Schizophrenia Diagnosis	40%	--	0%	100%
Service connected	26%	--	0%	67%

Culture and climate as predictors of LRC presence

Table 3 provides summary statistics for all variables. As an indicator of compliance with both LRC and Re-Engage directives, 134 of 158 sites (85%) reported having a designated LRC in place one week prior to Re-Engage rollout.

Table 4 presents odds ratios for binary logit models predicting initial compliance with the Re-Engage directive, via presence of an LRC, with measures of culture and climate. Across all models, higher values on culture or climate measures significantly increased the odds of having an LRC in place, after controlling for facility size and region. A one-unit increase in either entrepreneurial or hierarchical culture increased odds of early compliance by factors of 1.10 [95% confidence interval (CI): 1.03, 1.17] and 1.08 [CI: 1.03, 1.14], respectively. One-unit increases in either task or relational climate were of similar magnitude, increasing odds of early compliance by factors of 1.08 [CI: 1.01, 1.15] and 1.07 [CI: 1.01, 1.14], respectively.

Table 4.

Culture and climate as predictors of local recovery coordinator presence (odds ratios)

	CULTURE			CLIMATE
	Entrepre- neurial	Hierarch- ical	Composite	Task	Relational	Composite
Culture/climate measure	1.10** (1.03, 1.17)	1.08** (1.03, 1.14)	2.38** (1.41, 4.03)	1.07** (1.00, 1.14)	1.08** (1.01, 1.15)	1.75* (1.06, 2.88)
Facility size (000s)	1.01 (0.98, 1.03)	1.01 (0.98, 1.04)	1.01 (0.98, 1.04)	1.01 (0.99, 1.04)	1.01 (0.99, 1.04)	1.01 (0.99, 1.04)
Region (Ref=Northeast)
Midwest	0.20 (0.04, 1.05)	0.16* (0.03, 0.86)	0.19* (0.04, 0.98)	0.16* (0.03, 0.81)	0.18 (0.04, 0.92)	0.19* (0.03, 0.86)
South	0.51 (0.08, 3.08)	0.54 (0.09, 3.28)	0.59 (0.10, 3.60)	0.40 (0.07, 2.34)	0.41 (0.07, 2.41)	0.41 (0.07, 2.39)
West	0.25 (0.05, 1.37)	0.19 (0.04, 1.02)	0.21 (0.04, 1.16)	0.19* (0.03, 0.99)	0.17* (0.03, 0.91)	0.18* (0.03, 0.96)
N	158	158	158	158	158	158
AIC	123.61	123.62	121.76	129.40	128.38	128.77

^*p<0.05;

^**p<0.01

*** p<0.001

Note: Odds ratio indicate factor change in odds of having a designated Local Recovery Coordinator in place one week prior to Re-Engage roll-out. 95% confidence interval for odds ratio in parentheses

With respect to standardized measures of composite culture and climate, a standard deviation increase in composite climate increased odds of having an LRC by a factor of 1.77 [CI: 1.06, 2.88], all else equal. The impact of a standard deviation increase in composite culture was slightly stronger, increasing odds of having an LRC by a factor of 2.38 [CI: 1.41, 4.03], all else equal. Figure 2 presents predicted probabilities of initial policy compliance, via a designated LRC, for an average size VA in the Northeast at various levels of composite culture and climate. The predicted probability of having an LRC ranges from 22% at the lowest level of composite culture to a high of 98% at the highest level of composite culture.

Figure 2.

Predicted probabilities of designated Local Recovery Coordinator, by composite culture & climate

Note: Probabilities computed for a VA of average facility size in the Northeast region. Composite culture and climate measures were standardized. Range of composite culture and climate scores presented reflects distribution in data (see Table 3).

Culture, climate and LRC presence as predictors of implementation success

Six months after Re-Engage rollout, sites averaged updated documentation for 54% of listed Veterans with SMI lost to care, with a range from 0% to 100%. For sites that updated disposition for at least one Veteran, a second implementation outcome was examined: the percentage of listed Veterans with a completed contact (N=116). After six months, sites reported a mean completed contact rate of 33%, with range from 0% to 100%.

Tables 5 and 6 present results from linear regression models of culture and climate measures on Re-Engage compliance outcomes at six months. All models also include a measure of LRC presence. Table 5 shows results for site-level rate of updated disposition, for both separate and composite measure of culture and climate, and LRC presence, as well as controls for number of listed Veterans, region, and facility size. None of the measures of culture or climate were significant predictors of rate of updated disposition; however, presence of a dedicated LRC increased the rate of updated disposition by between 25 and 27 percentage points, all else equal. This effect was significant at p<0.05.

Table 5.

Linear regression of percent updated documentation, by culture, climate and LRC presence

	CULTURE			CLIMATE
	Entrepre- neurial	Hierarch- ical	Composite	Task	Relational	Composite
Culture/climate measure	−0.10 (−0.24)	0.19 (0.57)	0.80 (0.23)	−0.19 (−0.43)	−0.21 (−0.48)	−1.57 (−0.47)
Facility size (000s)	0.19 (1.11)	0.17 (1.05)	0.18 (1.05)	0.18 (1.06)	0.18 (1.07)	0.18 (1.07)
# Listed Veterans	−0.11 (−0.20)	−0.10 (−0.19)	−0.10 (−0.19)	−0.12 (−0.22)	−0.12 (−0.23)	−0.12 (−0.22)
Region (Ref=Northeast)
Midwest	−6.04 (−0.64)	−5.89 (−0.62)	−5.84 (−0.62)	−6.79 (−0.72)	−7.10 (−0.75)	−6.95 (−0.74)
South	3.43 (0.36)	4.79 (0.49)	4.12 (0.43)	2.39 (0.25)	2.42 (0.25)	2.38 (0.25)
West	−5.02 (−0.52)	−4.65 (−0.48)	−4.62 (−0.48)	−5.57 (−0.58)	−5.26 (−0.55)	−5.42 (−0.56)
Has LRC?	25.87** (2.64)	23.82* (2.53)	24.60* (2.50)	26.11** (2.73)	26.27** (2.74)	26.21* (2.73)
Intercept	27.51 (1.91)	29.06 (2.44)	28.52 (1.98)	28.11 (2.01)	28.54 (2.00)	28.57 (2.00)
N	158	158	158	158	158	158
R-squared	0.08	0.08	0.08	0.08	0.08	0.08
AIC	1171.39	1171.11	1171.40	1178.48	1178.43	1178.45

^*p<0.05;

^**p<0.01

*** p<0.001

Note: T-statistics in parentheses

Table 6.

Linear regression of percentage successfully contacted, by culture, climate, and LRC presence

	CULTURE			CLIMATE
	Entrepre- neurial	Hierarchi- cal	Composite	Task	Relational	Composite
Culture/climate measure	−0.44 (−1.38)	−0.35 (−1.37)	−4.03 (1.50)	0.36 (1.06)	0.45 (1.27)	3.20 (−1.50)
Facility size (000s)	−0.08 (−0.61)	−0.07 (−0.56)	−0.07 (−0.53)	−0.11 (−0.85)	−0.11 (−0.90)	−0.11 (−0.88)
# Listed Veterans	−0.83* (−2.11)	−0.84* (−2.13)	−0.84* (−2.14)	−0.76 (−1.91)	−0.74 (−1.87)	−0.76 (−1.90)
Region (Ref=Northeast)
Midwest	−14.93* (−2.11)	−14.57* (−1.97)	−14.98* (−2.02)	−10.43 (−1.40)	−8.92 (−1.17)	−9.69 (−1.29)
South	−5.03 (−0.71)	−6.07 (−0.83)	−6.09 (−0.84)	−0.37 (−0.05)	−0.34 (−0.05)	−0.30 (−0.04)
West	−18.25* (−2.24)	−17.25* (−2.15)	−18.13* (−2.24)	−11.60 (−1.45)	−11.89 (−1.50)	−11.69 (−1.47)
Has LRC?	0.78 (0.09)	0.07 (0.01)	0.77 (0.09)	−2.35 (−0.26)	−2.47 (−0.27)	−2.43 (−0.27)
% Black/African- American	−0.22 (−1.07)	−0.23 (−1.09)	−0.24 (−1.13)	−0.05 (−0.21)	−0.01 (−0.05)	−0.03 (−0.13)
% Homeless	−0.15 (−0.62)	−0.26 (−1.10)	−0.25 (−1.03)	−0.22 (−0.93)	−0.21 (−0.88)	−0.22 (−0.91)
% Schizophrenia Diagnosis	−0.01 (−0.01)	−0.06 (−0.29)	−0.04 (−0.18)	−0.09 (−0.45)	−0.11 (−0.55)	−0.10 (−0.50)
% Service- connected	0.05 (0.25)	0.07 (0.36)	0.06 (0.30)	0.04 (0.20)	0.05 (0.26)	0.04 (0.23)
Intercept	64.94 (4.11)	67.98 (4.29)	66.71 (4.23)	64.64 (4.04)	64.24 (4.02)	64.46 (4.03)
N	116	116	116	116	116	116
R-squared	0.12	0.12	0.13	0.10	0.11	0.10
AIC	770.475	770.502	770.083	780.649	780.096	780.349

Table 6 shows results for linear regression models predicting site-level rate of completed contacts after six months, for those sites who updated disposition for at least one listed Veteran (N=116). Again, none of the culture or climate measures were significant predictors of rate of completed contact. LRC presence also did not significantly affect rate of completed contacts, but LRC burden, or the number of Veterans an LRC was tasked with re-engaging, did have an effect, with each additional listed Veteran decreasing the rate of completed contacts by between 0.81 percentage points (in relational climate model) and 0.93 percentage points (in hierarchical culture model). Regional differences were also apparent, although to varying degrees, with VA facilities in the Midwest and West generally underperforming relative to their Northeastern counterparts. Southern VA sites were no different from Northeastern.

DISCUSSION

Policy mandates are one of the most powerful tools available for encouraging positive change in healthcare. Yet policy mandates do not ensure that change happens, nor than change happens in the same way. Recent reports on localized implementations of Affordable Care Act mandate have underscored this. The Department of Health and Human Services interim report to Congress on Medicaid Health Homes found significant variability across states, largely as states built their health home programs on top of existing coordination programs and infrastructures (Services, 2014).

Few studies have examined culture and climate as predictors of successful program implementation in real-life settings. Fewer still have examined culture and climate as predictors of implementation initiated by policy directives. In considering outcomes related to both VA policy compliance (via presence of LRCs responsible for implementing the Re-Engage program) and compliance after six months (via Re-Engage implementation) this paper provides new insight into the organizational barriers that hasten or impede compliance with a policy aimed at implementing a clinical initiative for patients with SMI.

The Re-Engage policy provided a unique opportunity to study implementation of a clinical initiative as: (1) implementation was mandated through a national policy, requiring all VA facilities to start implementation of Re-Engage at the same time; and (2) all VAs were provided with the same implementation support (via Replicating Effective Programs implementation strategy) and resources (via protected time for LRCs). In removing self-selection resource differentials, the rollout of the Re-Engage policy offered a rare window into how organizations with variable cultures and climates responded to the same mandate to change clinical practice for patients with SMI.

Notably, resource differentials still played a significant role in Re-Engage implementation. Although the LRC position predated the Re-Engage directive, LRCs were the centerpiece of the Re-Engage directive, The Re-Engage directive was significant for LRCs in that it explicitly reinforced their efforts on returning to care Veterans with SMI who had been lost to care. In mandating Re-Engage implementation, VA leadership also implicitly encouraged sites to make appropriate care for Veterans with SMI, and champions for such care, a priority. Nonetheless, 15% of sites did not have an LRC in place one week prior to program rollout.

Measures of culture and climate were significant predictors of LRC presence, with culture variables having slightly stronger effects than climate. Positive effects of both entrepreneurial culture and hierarchical culture, and the strong effect of composite culture, lend further weight to the idea that organizations with strong endorsements of both entrepreneurial and hierarchical cultures may be the most inclined to comply with policy encouraging change in clinical practice. With respect to this study, compliance required organizations to both respect VA leadership priorities of employing an LRC (hierarchical) and embrace implementation of new clinical practice (entrepreneurial). Unfortunately, the temporality of our LRC measure prevents us from disentangling whether organizational culture and climate measures were predictive of LRC presence as they represented long-term compliance with the 2006 mandate that first established the LRC role, or short-term compliance specifically the Re-Engage directive, or both. Future work should consider both whether such relationships exist, as well as what consequence they may be to furthering understanding of organizational commitment and implementation.

In turning to implementation outcomes, culture and climate measures were not significant predictors of implementation after six months. LRC presence, however, was a significant predictor of how well sites implemented the Re-Engage program, and specifically what percentage of listed Veterans with SMI lost to care had their documentation updated. Sites with LRCs in place had rates of updated documentation 25 percentage points higher than sites without. Looking at extremes, among facilities where no Veterans had updated documentation after six months, 60% had a designated LRC in place one week prior to Re-Engage roll out. By comparison, among facilities where all list Veterans had updated documentation after six months, 90% had a designated LRC. Thus a facility’s inability to appoint an LRC to carry out Re-Engage was associated with significantly poorer program implementation. This echoes prior work pointing to organizational commitment as prerequisite for successful recovery programs for individuals with SMI (Farkas et al, 2005).

Although the presence of an LRC predicted program implementation, it was not a significant predictor of completed contact. However, our models only assessed the percentage of completed contacts among facilities with updated documentation for at least one Veteran. Documentation of even one Veteran suggests that there was a clinician carrying out the Re-Engage program, even if not a designated LRC. The lack of a significant effect of having a designated LRC may indicate that facilities that designated a non-LRC clinician to carry out the Re-Engage program were as successful in contacting Veterans as facilities with LRCs carrying out the programs. Notably, the number of Veterans an LRC was tasked with contacting did have an effect on rate of completed contacts, with each additional listed Veterans decreasing the rate of completed contact by nearly 1 percentage point, net other factors.

Limitations of available data preclude more nuanced examinations of the impact of culture, climate and resources on Re-Engage implementation. Given the importance of LRCs to successful Re-Engage implementation, site-level data that described LRC assigned duties and ability to execute these duties may have exposed further important heterogeneity in both commitment to the LRC position and Re-Engage program, as well as the time available for the LRC to implement the Re-Engage program. More data on differing perceptions, priorities, and commitments of LRCs may have better explained differences in uptake amongst sites with a designated LRC. Further, predictive models that better explain why certain sites devote (or fail to devote) resources to staff responsible for implementing new programs would benefit understanding of organizational heterogeneity in implementation success.

Although our analyses here did not identify associations between organizational culture and climate and Re-Engage implementation, we remain enthusiastic for studies that better elucidate the pathways through which these organizational features impact implementation success. Our analyses regarding measures of culture and climate should be interpreted in the context of their limitations. First, we used site-aggregated data to develop our culture and climate measures. Ideal data would allow for measures of organizational culture and/or climate specific to clinics and units (e.g., mental health, primary care). Our inability to capture salient organizational differences specific to departments most impacted by Re-Engage likely downplayed the importance of culture and climate in predicting implementation. Further, general measures of culture and climate were examined, rather than measures of culture or climate specific to implementation—for example,. the Implementation Climate Scale (Ehrhart et al, 2014) or Implementation Leadership Scale (Aarons et al, 2014). These measures are likely to be more instructive for explaining heterogeneity in implementation success. Future implementation evaluations should consider collecting these metrics to identify priorities and pathways to improving implementation at sites that struggle. Further, and particularly under scenarios of policy-driven change, measures of change in organizational culture or climate may better predict successful implementation than baseline measures. Since we used data from the VA Annual Employee Survey to assess culture and climate, we were unable to assess frequent (e.g., monthly) changes in facility-level culture and climate as they related to Re-Engage implementation. Scientists may also want to consider using measures that capture such changes in readily comparable ways—for example, changes in employee turnover as an indicator of changes in climate. Finally, the implementation results presented here reflect outcomes after only six months, and under one specific implementation strategy—Replicating Effective Programs.

Implementation science has a key role to play in ensuring efficient and authentic translation of health policy into clinical practice, both by improving understanding of how pre-existing organizational characteristics facilitate and/or hinder successful implementation, but also by designing and tailoring implementation strategies in ways that accommodate these pre-existing characteristics. Implementation research has yet to investigate whether sites with different cultures or climate respond differently to different implementation interventions. The notion that “culture eats strategy for breakfast” is true only when strategy is not designed to exploit or cater to specific cultural strengths or deficiencies. Future research is needed to understand how organizational characteristics, including those discussed here, can be used to tailor provision of implementation strategies. In particular, work that explores potential time-varying effects of organizational characteristics and/or as moderators of comparative effectiveness of implementation strategies could help to inform efficient and effective implementation assistance.

CONCLUSION

Our findings based on a national policy implementation of a clinical program in the VA suggest that changes in health policy do not generally translate into homogenous change across health care systems. Frameworks and models of health care consistently point to organizational factors, like culture, climate and resources, as influential in determining the magnitude and tenor of localized change; yet little real-world evidence exists linking organizational differences to differences in policy implementation. Data from implementation of the VA Re-Engage policy provided a unique opportunity to examine the effect of organizational culture, climate and resources on real-world outcomes and policy uptake. Results confirmed that culture and climate were influential with respect to policy compliance (via presence of a designated LRC); however culture and climate were not associated with better implementation after six months. Rather, presence of an LRC responsible for implementing the program was the best predictor of Re-Engage implementation success after six months. More research in this vein can help implementation scientists understand optimal design and provision of implementation strategies.