The diffusion of evidence-based decision making among local health department practitioners in the United States

Abstract

Objectives

Evidence-based decision making (EBDM) is the process, in local health departments (LHDs) and other settings, of translating the best available scientific evidence into practice. Local health departments are more likely to be successful if they use evidence-based strategies. However, EBDM and use of evidence-based strategies by LHDs is not widespread. Drawing on Diffusion of Innovations theory, we sought to understand how LHD directors and program managers perceive the relative advantage, compatibility, simplicity, and testability of EBDM.

Design, Setting, and Participants

Directors and program managers in chronic disease, environmental health, and infectious disease from LHDs nationwide completed a survey including demographic information and questions about diffusion attributes (advantage, compatibility, simplicity, and testability) related to EBDM. Bivariate inferential tests were used to compare responses between directors and managers and to examine associations between participant characteristics and diffusion attributes.

Results

Relative advantage and compatibility scores were high for directors and managers, while simplicity and testability scores were lower. Although health department directors and managers in chronic disease generally had higher scores than other groups, there were few significant or large differences between directors and managers across the diffusion attributes. Larger jurisdiction population size was associated with higher relative advantage and compatibility scores for both directors and managers.

Conclusions

Overall, directors and managers were in strong agreement on the relative advantage of an LHD using EBDM, with directors in stronger agreement than managers. Perceived relative advantage has been demonstrated to be the most important factor in the rate of innovation adoption, suggesting an opportunity for directors to speed EBDM adoption. However, lower average scores across all groups for simplicity and testability may be hindering EBDM adoption. Recommended strategies for increasing perceived EBDM simplicity and testability are provided.

Introduction

A primary goal of local health departments (LHDs) is to assure that residents of their jurisdiction receive essential public health services.¹ Differences in LHD resources,³ organizational structures,² partners,³ and local health characteristics⁴ may influence the feasibility of providing services and help to determine which take priority. With 26.5% of LHDs reporting budget cuts in 2013, and 28.5% expecting budget cuts in 2014,⁵ it is critical LHDs devote existing resources to effective health promotion and disease prevention strategies.

Local health departments are more likely to be successful, and to meet accreditation standards, if they use evidence-based strategies.^6,7 Interventions not resulting in high health returns on investment can be costly to society, especially with the public health system facing increasingly limited financial resources.^7,8 Evidence-based decision making (EBDM) in public health is the process, in LHDs and other settings, of translating the best available scientific evidence about program and policy effectiveness into practice while taking into consideration local data, resources, and need.^7,9,10 The current national voluntary accreditation program under the Public Health Accreditation Board (PHAB) includes several domains that relate directly or indirectly to EBDM.⁶ Most directly, Domain 10 requires health departments to Contribute to and apply the evidence base of public health, by both identifying and using the best available evidence for making informed public health practice decisions, as well as promoting understanding and use of EBDM. In addition, several domains relate to the process of EBDM, including conducting community health assessments (Domain 1); identifying problems, recognizing community context (Domain 4); developing policies and plans (Domain 5); and evaluating health department processes, programs, and interventions (Domain 9). In addition Domain 11: Maintain administrative and management capacity, as well as Domain 8, Maintain a competent public health workforce, comprise Administrative-Evidence Based Practices.¹¹ Although informational resources such as the Guide to Community Preventive Services summarize existing evidence and provide specific recommendations for public health organizations, evidence-based decision making and use of evidence-based strategies by LHDs is not widespread.⁷

Barriers to adopting evidence-based strategies include political environment, a lack of relevant research, scarce resources, and characteristics associated with leadership and staff experience and expectations.^7,9,11,12 In addition to these informational and organizational barriers, effective dissemination of evidence-based strategies among LHDs is a significant barrier to adoption.¹² That is, while health professionals are likely aware of evidence-based guidelines, and agree with the idea of evidence-based strategies, relevant evidence is not always available to incorporate into public health programming.

Early work by Becker^13,14 and Mohr¹⁵ examining diffusion of innovation among LHD officers identified several organizational characteristics significantly associated with the diffusion process. Becker found that the earliest adopters of innovations varied by age and education and information seeking strategies used, while their jurisdictions varied by rurality. Those who would adopt less risky interventions were younger, in urban areas, recently graduated, and had a higher standing in their graduating class. Those who would take more risk and adopt less conventional interventions were older and more established in their leadership roles, in rural areas, and had an average standing in their graduating class. Mohr found that large health departments were more likely to innovate compared to small health departments, due to greater availability of staffing and funding to manage many programs.¹⁵ Within an organization, leadership and management are important actors in the adoption and implementation of innovations. Specifically, leadership often adopts innovations, while implementation of innovations relies strongly on the support of manager-level staff.¹⁶Adoption and implementation of an innovation by manager-level staff may be encouraged or hindered by aspects of organizational climate, including how decisions are made about adoption of innovative practices.^16,17 In organizations with highly centralized decision-making processes, managers may be excluded from participating in decision-making, which results in lower levels of implementation of innovations.

Rogers' diffusion of innovation (DOI) theory identifies five factors that influence diffusion and adoption of an innovative idea or strategy like EBDM.¹⁸ Specifically, to be adopted, an innovation must be perceived as offering relative advantage, being compatible, being simple, being testable, and being observable.¹⁸ Relative advantage is “the degree to which an innovation is perceived as better than the idea it supersedes.” Compatibility is “the degree to which an innovation is perceived as consistent with existing values, past experiences, and needs of potential adopters.” Simplicity is “the degree to which an innovation is perceived as relatively difficult to understand.” Testability (or trialability) is “the degree to which an innovation may be experimented with on a limited basis.”¹⁸ These factors are important at the second stage of the adoption process, the persuasion stage, where an individual has already been exposed to a new idea and is seeking information about the idea in order to move to the next stage and make a decision on adoption.

With the importance of EBDM for effective public health practice, we sought to better understand how LHD directors and program managers perceive the relative advantage, compatibility, simplicity, and testability of EBDM. This study builds on what we know from past research on diffusion and recent work by Erwin and colleagues,¹⁹ which described the importance of individual and organizational factors in explaining performance of administrative evidence-based practices by LHD directors and managers. Specifically, we examined and compared LHD directors and managers from three program areas (chronic disease, environmental health, and infectious disease) on their perceptions of EBDM and examined associations between perceptions and LHD jurisdiction population size, director/manager age, and director/manager education level.

Methods

Sampling

The study team developed, tested, and validated a survey instrument to examine EBDM in LHDs as part of the LEAD (Local Evidence for Affecting Decisions) Public Health Project to identify local evidence for effecting decisions about public health (see http://prcstl.wustl.edu/ResearchAndFindings/Pages/LEAD-public-health-project.aspx). For the analyses reported here, and in Erwin's recent study of administrative evidence-based practices,¹⁹ a stratified random sample of US LHDs was drawn from the database of the National Association of County and City Health Officials. Health departments were drawn from five groups, according to jurisdiction population size of a LHD: <25,000, 25,000-49,999, 50,000-99,999, 100,000-499,999, and 500,000+ persons. A sample of 1,067 LHDs was used as the initial sample. Next, an online survey was delivered nationally to email accounts of 1,067 LHD directors. After excluding non-valid email addresses the final sample was 967. Five-hundred thirty-one LHD directors responded to the survey; 14 refused to complete the survey, leaving 517 valid responses (54% response rate); 332 program managers from three program areas also participated. Program managers were from 196 LHDs and represented chronic disease (n=99), infectious disease (n=113), and environmental health (n=100) (response rates: 59%, 64%, and 56%, respectively). We sent 4 email reminders and made 2 phone calls to LHD directors and sent 3 email reminders to program managers. Several LHDs had two or more program manager participants; to ensure independent observations; one program manager from each of these LHDs was randomly selected for analysis, for a total of 196 program managers matched with 196 LHD directors. The study was approved by the Institutional Review Board at Washington University in St. Louis.

We used chi-squared to compare non-responders and responders across jurisdiction population size, governance type, and region. We found no significant association between response and governance type, however, we found a significant association between response and population category (χ²(4)=10.47; p=.03), with standardized residuals indicating that practitioners from health departments serving fewer than 25 thousand constituents were less likely than expected to respond to the survey. Specifically, the response rates were 41.0%, 50.5%, 51.4%, 52.7%, and 52.3% for the five jurisdiction population categories in order from smallest to largest: <25,000, 25,000-49,999, 50,000-99,999, 100,000-499,999, and 500,000+. Likewise, a significant association between response and region (χ²(3)=28.04; p<.001) with standardized residuals indicating that fewer than expected practitioners from LHDs in the Northeast, but more than expected in the Midwest, responded.

Diffusion items

Ten statements were made about the four areas of diffusion (see Table 1); LHD directors and program managers were asked to indicate how strongly they agreed with each of the ten statements using a seven-point scale from strongly disagree to strongly agree. The instrument is available on the Prevention Research Center in St. Louis website (see LEAD project URL above). The higher the score, the more strongly the participant agreed with the statement. Note that, for statements 8 and 9 in Table 1, a higher score indicated that the director or manager believed many changes or additional staffing would be needed to implement EBDM. Higher scores on all other items indicate support for EBDM without major changes. We computed scale scores for compatibility, simplicity, and testability by taking the mean of the items comprising the score; items 8 and 9 were reverse coded prior to computing scale scores to make directionality similar for all items.

Table 1.

Survey items for measuring agreement with diffusion characteristics in local health department directors and managers.

Diffusion characteristic	Survey item(s)
Relative advantage	(1) Positive changes in my agency can occur as a result of EBDM
Compatibility	(2) EBDM fits with my agency's mission or goals
	(3) EBDM is consistent with what we're already doing as an agency
Simplicity	(4) In my agency, EBDM is easy to understand
	(5) In my agency, EBDM is easy to implement
Testability	(6) Components of an EBDM process can be subdivided in order to implement them
	(7) EBDM has minimal financial cost to implement
	(8) We would have to make many changes in my agency to implement EBDM
	(9) We would have to hire new staff or retrain staff to implement EBDM
	(10) Components of EBDM can be implemented or tried without fully committing to the entire process

Data analysis

A sign test was used to determine whether directors and managers from the same LHD differed with respect to age and education. Although diffusion characteristics were measured on ordinal scales, responses, and scales resulting from averaging responses, were normally distributed, with no large skew or kurtosis identified. Given the large number of categories and the relatively normal distribution, paired t-tests were used to determine whether there was an association between the scores of directors and program managers overall and from each of the three areas in the same LHD. For each significant paired t-test, a measure of effect size (Cohen's d) was calculated to determine the magnitude of the difference. In addition, associations between diffusion characteristics and LHD jurisdiction size, and between diffusion characteristics and individual characteristics of age and education categories were examined using Spearman's rho rank-order correlation coefficient and one-way analysis of variance for both directors and managers. Due to the large number of significance tests, we adjusted the alpha for all analyses to.01; results with a p-value at or below.01 were considered statistically significant.

Results

Sample characteristics

The median jurisdiction population size for the 196 LHDs included in this analysis was 66.5 thousand with a range of 950 to 2.2 million. Age and education were different between directors and managers, with more directors being in older age categories and higher degree categories compared to managers (Table 2).

Table 2.

Comparison of age and education of directors and program managers in 196 local health departments nationwide.

			Program managers
	Directors n (%)	Managers n (%)	Chronic disease n (%)	Environ health n (%)	Infectious disease n (%)
Age category
20-29	4 (2.1)	9 (4.7)	2 (3.0)	3 (4.5)	4 (6.9)
30-39	24 (12.4)	41 (21.5)	17 (25.8)	14 (20.9)	10 (17.2)
40-49	40 (20.7)	47 (24.0)	14 (21.2	24 (35.8)	9 (15.5)
50-59	79 (40.9)	68 (35.6)	26 (39.4)	18 (26.9)	24 (41.4)
60+	46 (23.8)	26 (13.6)	7 (10.6)	8 (11.9)	11 (19.0)
Degree category
Bachelors or lower	39 (19.9)	75 (38.3)	19 (28.4)	45 (65.2)	11 (18.3)
Nursing	24 (12.2)	46 (23.5)	15 (22.4)	1 (1.4)	30 (50.0)
Other master's	59 (30.1)	36 (18.4)	18 (26.9)	13 (18.8)	5 (8.3)
Public health master's	33 (16.8)	27 (13.8)	8 (11.9)	8 (11.6)	11 (18.3)
Doctoral	41 (20.9)	12 (6.1)	7 (10.4)	2 (2.9)	3 (5.0)

Relative advantage

There was a significant difference with a small-to-medium effect size (t=4.60; p<.001; d=.35) between the average relative advantage score for directors (m=6.22; sd=.79) and managers overall (m=5.83; sd=.99) in the same LHD (Figure 1). There were no significant difference between directors and managers in chronic disease (m=6.07; sd=.76), a significant difference with a medium effect size (t=3.63; p=.001; d=.48) between directors and managers in environmental health (m=5.60; sd=1.15), and a significant difference with medium effect size (t=2.78; p=.008; d=.39) for directors and managers in infectious disease (m=5.81; sd=.99). There was also a positive and significant moderate correlation between jurisdiction population size and relative advantage for directors (r=.22; p=.002), managers overall (r=.25; p=.001), and chronic disease managers (r=.34; p=.008), but not for environmental health or infectious disease managers.

Figure 1.

Average LHD director and manager scores for paired data on the 10 diffusion items. Significant differences between directors and managers marked by ^*.

Age was not significantly associated with relative advantage score for directors or managers of any type. There was a significant association between education and relative advantage score for directors (F=5.10; p=.001) but not for managers of any type. Bonferroni post-hoc tests found that directors with a Bachelor's or lower scored significantly lower than those having other master's degrees.

Compatibility

There was no significant difference between the average score for directors (m=5.54; sd=.97) and managers overall (m=5.31; sd=1.14) in the same LHD for compatibility. However, while managers in chronic and infectious disease did not differ significantly from their directors on compatibility perceptions, compared to directors in the same LHD, managers in environmental health (m=5.08; sd=1.22) had significantly lower compatibility scores (t=2.76; p=.008; d=.36).

Population size had a positive and significant associations with compatibility for directors (r=.18; p=.01), managers in general (r=.21; p=.005), and environmental health managers (r=.36; p=.005). Age and education had no significant association with score for compatibility for directors or any type of manager.

Simplicity

There were no statistically significant differences between the average score for directors and managers of all types in the same LHD for simplicity. Population size, age, and education had no significant association with scores for simplicity for directors or managers.

Testability

There was a significant difference with a small effect size (t=2.74; p=.007; d=.21) in average testability scores between directors (m=4.44; sd=.75) and managers (m=4.25; sd=.74), however, there were no significant differences in testability scores between directors and managers when each of the three program-related groups (i.e., chronic, infectious, environmental) was examined separately. Population size, age, and education had no significant association with scores for testability for directors or managers, with the exception of a positive and significant correlation between population size and testability for environmental health managers (r=.35; p=.007).

Discussion

Overall, both directors and managers were in strong agreement that using EBDM would result in positive change, with directors in significantly stronger agreement than managers. Directors and manager scores were similar with one another regarding the compatibility, simplicity, and testability of EBDM, however, the scores for these elements of diffusion were lower than for relative advantage. Responses regarding simplicity and testability were lowest and close to neutral near the mid-point of 4 (“Neither agree nor disagree”) on the 7-point scale. Most differences between directors and managers were not statistically significant; however, consistent with chronic disease managers scoring highest in the use of administrative evidence-based practice in a related study,¹⁹ managers in chronic disease had consistently higher scores compared to other manager types across all diffusion areas, and were most consistent with LHD directors. Managers in environmental health had consistently lower scores than other managers and LHD directors.

Consistent with early diffusion work by Mohr¹⁶ and recent findings from Erwin et al., ¹⁹ we found significant positive associations between jurisdiction population size and EBDM diffusion characteristics (i.e., larger jurisdiction was correlated with larger scores) for directors (relative advantage, compatibility), managers overall (relative advantage, compatibility), chronic disease managers (relative advantage), and environmental health managers (testability). However, contrary to early diffusion studies^13-15 the individual-level characteristics of age and education level had little to no influence on diffusion attributes.

Perceived relative advantage has been demonstrated to be the most important of the factors in predicting the rate of innovation adoption,^18,20,21 so high perceived relative advantage is promising for the spread of EBDM in LHDs. In addition, the significant difference in relative advantage scores between directors and managers suggests two possible strategies for improving the diffusion of EBDM: (1) increase encouragement by leadership for EBDM; and (2) increase the connections among managers in larger LHDs and their counterparts in smaller LHDs. Leaders who devote personal influence to encourage EBDM adoption by managers and who work to create a climate supportive of innovation by managers and other professional staff may be able to increase innovation in their organizations.^16,17,20 Specifically, because LHD directors have significantly higher relative advantage scores, they could work to communicate the relative advantage of EBDM and foster a climate of innovation supportive of EBDM across the LHD by including managers in the decision to adopt EBDM. Our results suggest it may be especially important for leadership to increase communication about the advantages of EBDM in smaller LHDs, particularly with environmental health managers. In addition, peer networks may also influence diffusion. A recent study of communication ties among LHDs found that ties associated with diffusion were more likely between LHDs sharing the same population size, staffing and funding levels, and programming types.²² With higher relative advantage scores in larger LHDs, it might be useful to develop new communication ties between LHD managers in smaller LHDs and their peers at larger LHDs.

Simplicity^23,24 and testability²⁵ are also strongly associated with innovation adoption;^17,26 low scores in simplicity and testability across leadership and management in LHDs may therefore be barriers to EBDM diffusion. Breaking an innovation down into manageable parts for incremental adoption can increase simplicity;²⁵ as can demonstrating the innovation.²⁶ Perceived testability is greater if testing can be done slowly or on a small scale.¹⁷ Breaking EBDM into parts or implementing EBDM on a small scale may require partnerships between researchers and practitioners to develop feasible strategies that maintain innovation fidelity.²⁷

Limitations

This study is limited by cross-sectional self-reported data. In addition, the response rates for LHD directors (54%) and program managers (59%, 64%, and 56% for the three programs) were somewhat low and may bias the results. We also had fewer participants than expected from very small and Northeastern health departments and more than expected from Midwestern health departments represented, which may have influenced our results. Despite these limitations, this study is the first that we know of to examine diffusion attributes for EBDM in a range of LHD settings.

Use of EBDM aids in ensuring that LHDs are selecting effective strategies with the best public health return on investment. Although use of EBDM is not yet widespread across LHDs, our results indicate that directors and program managers in LHDs generally agree that EBDM has relative advantage over other strategies. With past research demonstrating that relative advantage is the most important of the diffusion factors examined here, our findings bode well for EBDM diffusion across LHDs. Strategies to increase perceived relative advantage among managers and perceived simplicity and testability among all groups may increase the likelihood of EBDM adoption, although recognizing that diffusion of an innovation like EBDM is often slow, even when it is perceived as having high relative advantage and strategies to encourage diffusion are employed.²⁰