Understanding administrative evidence-based practices: Findings from a survey of local health department leaders

Abstract

Background

There are sparse data showing the extent to which evidence-based public health is occurring among local health departments.

Purpose

The purpose of the study was to describe the patterns and predictors of administrative evidence-based practices (structures and activities that are associated with performance measures) in a representative sample of local health departments in the United States.

Methods

A cross-sectional study of 517 local health department directors was conducted from October through December 2012 (analysis in January through March 2013). The questions on administrative evidence-based practices included 19 items based on a recent literature review (five broad domains: workforce development, leadership, organizational climate and culture, relationships and partnerships, financial processes).

Results

There was a wide range in performance among the 19 individual administrative evidence-based practices, ranging from 35% for access to current information on evidence-based practices to 96% for funding via a variety of sources Among the five domains, values were generally lowest for organizational climate and culture (mean for the domain = 49.9%) and highest for relationships and partnerships (mean for the domain = 77.1%). Variables associated with attaining the highest tertile of administrative evidence-based practices included having a population jurisdiction of 25,000 or larger (adjusted odds ratios (aORs) ranging from 4.4 to 7.5) and state governance structure (aOR=3.1).

Conclusions

This report on the patterns and predictors of administrative evidence-based practices in health departments begins to provide information on gaps and areas for improvement that can be linked with ongoing quality improvement processes.

INTRODUCTION

There have been substantial improvements in health and longevity over the past century in the United States and other developed countries. This has come in part from the implementation of public health programs and policies, covering a wide range of issues including workplace safety, immunizations, tobacco control, healthier eating, and reductions in motor vehicle crashes.^1–4 A focus on evidence-based public health (EBPH) has been described as the integration of science-based interventions with community preferences to improve the health of populations.⁵ The importance of a stronger focus on EBPH is highlighted in numerous publications, including the Public Health Accreditation Board Standards that seek to “contribute to and apply the evidence base of public health”⁶ as well as authoritative reports from the Institute of Medicine that recommend specific actions to improve public health practice.^{7, 8} These publications and guidelines highlight the importance of using the best available evidence and the role of health departments in adding to the body of evidence on how to improve population health. There also is now a considerable literature on the barriers to EBPH (e.g., lack of time/competing demands, inadequate funding/high cost, absence of organizational support).^9–14

Overcoming these barriers and thus, fostering EBPH requires a combination of applying evidence-based interventions from scientific sources (e.g., the Community Guide,¹⁵ the Cochrane Collaboration¹⁶) along with performance in carrying out effective organizational practices in health departments or other agencies. This process of EBPH can include so-called “administrative evidence-based practices (A-EBPs),” which are agency (health department)- and work unit-level structures and activities that are positively associated with performance measures (e.g., achieving core public health functions, carrying out evidence-based interventions).¹⁷ These A-EBPs often fit under the umbrella of public health services and systems research,^{18, 19} and cover five major domains of workforce development, leadership, organizational climate and culture, relationships and partnerships, and financial processes. These practices were recently articulated via a literature review¹⁷ and are potentially modifiable within a few years, making them useful targets for quality improvement efforts.^20–23

While expert reports, state and local quality improvement efforts, and accreditation processes are drawing increasing attention to A-EBPs,^24–29 there are sparse data showing the extent to which performance in EBPH is occurring among local health departments (LHDs). To fill this gap, the current study had two objectives: 1) describe patterns in A-EBPs from a nationally-representative sample of LHDs; and 2) describe predictors of A-EBPs according to characteristics of the individual LHD leader and the characteristics of the LHD.

METHODS

Sample

A stratified random sample of US LHDs was drawn from the data base of the National Association of County and City Health Officials (NACCHO).³⁰ Health departments were drawn from five groups, according to jurisdiction size (i.e., population service area) of a LHD: <25,000, 25,000–49,999, 50,000–99,999, 100,000–499,999, and 500,000+ persons. A sample of 1,067 LHDs was used as the initial sample, out of a total of 2,565 LHDs. Human subjects approval was obtained from the Washington University Institutional Review Board.

Questionnaire development and testing

The survey instrument was based in part on a logic model adapted from PHSSR frameworks^31–34 and previous work with state and local health departments, where standardized questions existed.^{12, 35–39} The questionnaire included six sections (i.e., biographical data, A-EBPs, diffusion attributes, barriers to EBPH, use of resources, competencies in EBPH) and 66 questions. The A-EBPs section of the instrument included 19 questions that were new and based on findings from a recent literature review.¹⁷ Six A-EBPs questions used a dichotomous response (yes or no) and 13 used a 7-point, Likert-scaled. The instrument was designed for completion in 15 minutes or less.

The survey was reviewed by the core research team (n = 11) and experts at NACCHO (n = 2). After three rounds of revision, the instrument underwent cognitive response testing with 12 experts who were representative of the target audience (LHD directors). In these cognitive methods,^40–43 testing determined: 1) question comprehension (e.g., What does the respondent think the question is asking?); 2) information retrieval (e.g., What information does the respondent need to recall from memory in order to answer the question?); and 3) decision processing (e.g., How do they choose their answer?). The research team incorporated cognitive response feedback into a further revision of the survey. In the next round of instrument development, a group of 38 LHD practitioners were sampled as part of a test-retest study to improve the instrument. These data are reported elsewhere,⁴⁴ and showed that the majority of items (41/54=76%) had substantial to nearly perfect reliability and no items had poor reliability.^{44, 45} For the A-EBPs questions, Cronbach’s alpha values ranged from 0.67 to 0.94.⁴⁶

Data collection

Data were collected using an online survey (Qualtrics software⁴⁷) that was delivered nationally to email accounts of 1,067 LHD directors. One person was invited from each selected LHD. The survey was opened for a 2-month time period (from October through December 2012) in which 4 email reminders and 2 rounds of phone calls were delivered to bolster response rate. After excluding non-valid email addresses from the initial sample, the final recruitment sample was 967. There were 517 valid responses to the survey (54% response rate). Health department characteristics were similar between respondents and non-respondents except for two variables. Respondents were less likely to work in health departments with jurisdictions of less than 25,000 people (26.2% compared with 35.3%) and were less likely to reside in the northeast (16.9% compared with 30.5%). The median survey administration time was 14 minutes.

Data analysis

Descriptive statistics were calculated for each A-EBP. The sample characteristics were derived from the survey data (individual characteristics in Table 1) and from archival data for each health department using US Census regions, population size of jurisdiction, and LHD governance structure.⁴⁸ Sample sizes varied due to missing data. For each of five A-EBP domains and for all A-EBPs combined, the scores were summed, ranked, and placed into tertiles. Using unconditional logistic regression models, odds ratios (ORs) and 95% confidence intervals (CIs) were calculated to compare those who reported the highest third of A-EBPs scores with those who reported the lowest third. For the final model, significant variables and covariates that contributed to the fit of the model were retained, allowing us to calculate adjusted ORs. In addition, state was included as a covariate since LHDs are nested in states and there may be some clustering effect.

Table 1.

Characteristics of the sample of local health departments, United States, 2012

Characteristic	No.	Percent
Individual
Age (yrs)
20–39	52	10.0
40–49	110	21.3
50–59	228	44.1
60 and older	127	24.6
Gender
Female	315	60.9
Male	202	39.1
Job Position
Top executive, health officer, commissioner	351	67.9
Administrator, deputy, or assistant director	117	22.6
Manager of a division or program, other	49	9.4
Highest Degree
Doctoral	91	17.7
Master of Public Health	88	17.1
Other masters degree	138	26.8
Nursing	97	18.8
Bachelors degree or less	101	19.6
Health Department
Census Region
Northeast	87	16.9
Midwest	200	38.8
South	149	28.9
West	80	15.5
Population of Jurisdiction
<25,000	135	26.2
25,000 to 49,999	110	21.4
50,000 to 99,999	95	18.4
100,000 to 499,999	106	20.6
500,000 or larger	69	13.4
Governance Structure
State governed	51	9.9
Locally governed	416	80.8
Shared governance	48	9.3

RESULTS

From the 517 LHDs that participated in the survey, two-thirds of respondents were the top health official in the health departments, followed by a deputy or assistant director (23%) (Table 1). Most of the respondents were over 50 years of age and were female. Respondents had a variety of educational backgrounds. Approximately one-fourth of respondents held a (non-MPH) masters degree, 17% held an MPH, 18% held a doctorate (e.g., PhD, DrPH, MD), and 20% held a bachelors degree or less. The largest proportion of respondents was drawn from the Midwest (39%) and South (29%). As designed in the sampling scheme, health departments represented all jurisdiction size categories. Most health departments were locally governed (81%), with about 10% classified as state governed and 9% with shared state/local governance.

There was a wide range in performance among the 19 individual A-EBPs, ranging from 35% for access to current information on EBPH practices to 96% for funding via a variety of sources (Table 2). Among the five broad domains (workforce development, leadership, organizational climate and culture, relationships and partnerships, financial processes), values were generally lowest for organizational climate and culture with three of four items reported by 42% to 43% of LHDs while 71% of LHDs reported presence of the fourth item, promotion of lifelong learning (mean for the domain = 49.9%). Only 4 of 19 items were reported as present among more than 75% of LHDs. Five of 19 items were reported by fewer than 50% of the LHDs. The second lowest scoring domain was leadership (mean for the domain = 56.8%).

Table 2.

Administrative evidence-based practices in local health departments, United States 2012

Administrative Practice	No.	Percent
Workforce Development
Access to training in:
Quality improvement processesa	418	82.1
Performance assessmenta	368	71.5
Management practicesa	361	70.0
Evidence-based decision making (EBDM)a	279	59.0
Access to current information on EBDM processesb	181	35.0
Average for domain		63.5
Leadership
Foster staff participation in decision makingb	432	83.6
Encourage use of EBDMb	311	60.2
Ability to lead in EBDMb	271	52.4
Hire people with experience in public healthb	269	52.0
Hire people with public health degreeb	184	35.6
Average for domain		56.8
Organizational Climate and Culture of Agency
Promotes life-long learningb	367	71.0
Access to EBDM information relevant to community needsb	224	43.3
Access to current research evidenceb	222	42.9
Culture that supports EBDMb	218	42.2
Average for domain		49.9
Relationships and Partnerships
Important to develop partnerships with both health and other sectorsb	477	92.3
Partnerships have missions that align with agencyb	365	70.7
Important to have partners who share resourcesb	353	68.3
Average for domain		77.1
Financial Characteristics of Agency
Funded through a variety of sourcesa	159	95.8
Allocated resources for quality improvementa	282	54.5
Average for domain		75.2

^aDichtomous (yes/no) response option.

^b7-point Likert-scale response option; frequency shown is those who “strongly agree” and “agree.”

In unadjusted, bivariate analyses, an array of variables predicted attainment of the highest tertile of A-EBPs (Table 3). Since findings were similar by domain, the summary results are presented. After adjustment for all statistically significant bivariate predictors, a few variables were associated with attaining the highest tertile of A-EBPs, including age of 50 to 59 years (adjusted odds ratio (aOR) = 2.5; 95% CI = 1.08, 6.0), population jurisdiction of 25,000 or larger (aORs ranging from 4.4 to 7.5) and state governance structure (aOR = 3.1; 95% CI = 1.04, 9.1).

Table 3.

Predictors of administrative evidence-based practices,^a United States, 2012

Characteristic	No. in highest tertile	No. in lowest tertile	Unadjusted odds ratio (95% confidence interval)	Adjusted odds ratiob (95% confidence interval)
Individual
Age (yrs)
20–39	13	24	1.0	1.0
40–49	33	32	1.9 (0.8, 4.4)	1.5 (0.6, 3.9)
50–59	77	61	2.3 (1.1, 5.0)	2.5 (1.08, 6.0)
60 and older	37	40	1.7 (0.8, 3,8)	1.5 (0.6, 3.7)
Gender
Female	96	95	0.98 (0.6, 1.5)	--
Male	64	62	1.0	--
Job Position
Top executive, health officer, commissioner	119	97	1.6 (0.7, 3.5)	--
Administrator, deputy, or assistant director	28	43	0.9 (0.4, 2.0)	--
Manager of a division or program, other	13	17	1.0	--
Highest Degree
Doctoral	39	24	3.1 (1.5, 6.4)	2.1 (0.9, 5.3)
Master of Public Health	28	27	2.0 (0.96, 4.1)	1.9 (0.8, 4.6)
Other masters degree	50	33	2.9 (1.5, 5.7)	1.9 (0.9, 4.1)
Nursing	20	28	1.4 (0.6, 2.9	1.5 (0.6, 3.6)
Bachelors degree or less	23	44	1.0	1.0
Health Department
Census Region
Northeast	19	40	1.0	1.0
Midwest	55	61	1.9 (0.98, 3.7)	1.4 (0.6, 3.0)
South	65	30	4.5 (2.3, 9.2)	1.9 (0.8, 4.8)
West	21	26	1.7 (0.8, 3.8)	1.5 (0.6, 3.6)
Population of Jurisdiction
<25,000	14	64	1.0	1.0
25,000 to 49,999	40	26	7.0 (3.3, 15.0)	7.5 (3.3, 17.3)
50,000 to 99,999	35	28	5.7 (2.7, 12.2)	4.9 (2.1, 11.2)
100,000 to 499,999	46	23	9.1 (4.3, 19.6)	7.1 (3.0, 16.9)
500,000 or larger	25	16	7.1 (3.0, 16.8)	4.4 (1.6, 12.5)
Governance Structure
State governed	24	9	3.3 (1.5, 7.4)	3.1 (1.04, 9.1)
Locally governed	114	141	1.0	1.0
Shared governance	22	7	3.9 (1.6, 9.4)	2.5 (0.8, 7.6)

^aThe administrative evidence-based practices (A-EBP) summary score was calculated by summing the respondents’ rankings for the 19 individual questions into a summary score then placing the summary scores into tertiles.

^bThose variables that were significant in unadjusted analyses (i.e., age, highest degree, census region, population of jurisdiction, governance structure) were retained in the final model to calculate adjusted odds ratios. The odds ratios represent the odds of being in the highest tertile.

Because jurisdiction size was the most robust predictor of A-EBP performance, the performance on 19 individual A-EBPs was compared in LHDs serving fewer than 25,000 people with LHDs serving 25,000 or more people (not shown in table). Across the 19 A-EBPs, smaller LHDs showed lower performance on all but one A-EBP (promotes lifelong learning). The largest differences between smaller and larger LHDs were shown for four specific A-EBPs: access to current information on EBPH practices (relative difference [i.e., the higher value minus the lower value divided by the higher value] = 49%), hire people with public health experience (relative difference = 49%), hire people with public health degree (relative difference = 62%), and access to current research evidence (relative difference = 51%). In addition, these smaller LHDs were three times more likely to be led by someone holding a nursing degree (36.6% compared with 12.7%).

DISCUSSION

While the importance of applying principles of EBPH in public health practice has become more prominent over the past 15 years,^49–54 most inquiry has focused on the uptake of evidence-based interventions often providing sparse information on how these concepts are applied in local public health practice (the performance in EBPH). This study provides the first nationwide data on a broad range of A-EBPs among LHDs in the United States. While the current study focused on the United States, the same principles of EBPH are important in other regions of the world.^55–60 Across the five domains of A-EBPs, a wide range of estimates was found from the lowest performance in A-EBPs related to organizational climate/culture and the highest responses for partnership development and agency support via multiple funding streams.

Administration and management capacity is 1 of 12 LHD accreditation domains established by the Public Health Administration Board.²⁶ The A-EBPs identified in the previous review,¹⁷ now measured in the current study, can be linked with LHD performance, quality improvement, and accreditation processes.^{20, 21, 24, 25, 61–64} Similar to the current findings, national data on LHDs show that health departments with large size of jurisdiction and centralized governance are more likely to engage in quality improvement activities, provide training, and have trained managers.^{21, 48, 65}

The current findings highlight the need to focus more strongly on enhancing the climate and culture for EBPH in LHDs. The health-related literature on climate and culture comes largely from studies of health care organizations (e.g., hospitals) and mental health service organizations. In these settings, organizational culture is the degree to which employees perceive an honest, fair, and trusting workplace.^{66, 67} Organizational climate is related to the localized manifestation of the culture, can vary across teams or units, and is often less stable over time compared with culture.^66–68 The few studies of public health practitioners suggest difficulty in changing organizational climate and culture.^{10, 12} Related to climate and culture, there also are numerous studies showing the linkage between health department leadership and EBPH (e.g., leaders who foster a climate supportive of EBPH).^{9, 39, 69–71} There are now leadership training programs to foster leadership skills and develop a culture for EBPH.^72–74 It is also likely that even in the presence of committed leadership, a “critical mass” of committed staff and a social network in support of EBPH are needed.^{75, 76}

Strengthening EBPH competencies needs to take into account the diverse education and training backgrounds of the workforce. The emphasis on principles of EBPH is not uniformly taught across disciplines that comprise the public health workforce, in part because most people working in day-to-day public health practice lack formal training in core public health disciplines.^{51, 77} Several approaches in the literature show promise for addressing the deficits in EBPH-related skills.^{38, 78} One promising approach involves the use of knowledge brokers (i.e., a masters-trained individual available for technical assistance). Used more in Canada than in the United States, a knowledge broker provides a link between research and end users (practitioners) by developing a mutual understanding of goals and cultures, collaborating with end users to identify issues and problems for which solutions are required, and enhancing access and use of research evidence in practice and policy.⁷⁸ Although there are few well-conducted evaluations of knowledge broker impact,⁷⁹ there is considerable evidence of effectiveness in other fields, particularly from business and agricultural sectors.^80–82 Implementation of training and capacity building to address A-EBPs should take into account principles of adult learning (e.g., respect the experience of learners, conduct active learning).^{83, 84}

These findings also suggest that a “one size fits all” approach for improving A-EBPs may not be effective. The sharp differences between smaller and larger LHDs in performance in A-EBPs highlight the challenges in delivering effective public health services in rural settings.^85–88 Realistic expectations for smaller LHDs can be linked with the recent recommendation from the Institute of Medicine calling for a minimum set of services that no health department should be without.⁷ These cover both foundational capabilities (e.g., surveillance, policy development capacity, quality improvement) and basic programs (e.g., mainly categorical programs: maternal and child health promotion, communicable disease control, chronic disease prevention). The A-EBPs fit most closely with the foundational capabilities and provide baseline data and a reliable method for measuring administrative and management capacity. Smaller LHDs were also much more likely to be led by a person trained in nursing. While evidence-based practice has been prominent in nursing training for decades, it has largely focused on a patient orientation similar to evidence-based medicine.^{89, 90} Broadening training for nurses (both in formal, degree-based education and in continuing education) to focus more on public health sciences and skills may benefit the uptake of A-EBPs in LHDs.^{91, 92} While it is not intuitively obvious how smaller LHDs could modify the predictor status of their small jurisdictions, there is increasing evidence that cross-jurisdictional sharing and regionalization of multiple LHDs may provide opportunities to enhance A-EBPs in such settings.^{28, 93}

Access to information also seems to be a particular challenge for smaller LHDs since in the current sample, smaller health departments were half as likely to have access to current research evidence for public health. Internet connectivity alone does not ensure access to research, as many journals do not provide free access; however, a recent decision to promote access to the new on-line journal Frontiers in Public Health Systems and Services Research to all LHDs through NACCHO is just one example of efforts which could enhance performance in A-EBPs.[personal communication; B. Pestronk & G. Mays, April 2013] A recent systematic review suggests that in order to increase use of and access to scientific evidence in public health, two-way communication is needed between practitioners and researchers.¹⁴ This may be enhanced by practice-academic linkages,^{94, 95} yet may be particularly challenging for widely dispersed LHDs that are not well linked with universities. Recent experiences of a select number of LHDs serving as demonstration sites for conducting community health assessments, however, indicate that there is a wide range of academic institutions—including community colleges and other institutions which do not have schools or programs in public health—with which LHDs can successfully partner.⁹⁶ There is a need for creating new and creative methods for reaching LHDs—this may rely on more effective use of opinion leaders,⁹⁷ social media,⁹⁸ organizational partnerships, and new priorities from funding agencies (to better design research for dissemination).^99–102

A few limitations of this study should be noted. The main limitation is that the data are self-reported. Although psychometric testing of the instrument showed it is reliable, it is difficult to precisely ascertain the difference between people’s report of A-EBPs and how these practices are being carried out in their agency (validity). In addition, the response rate was modest at 54%, suggesting the possibility of response bias due to lower representation from health departments with jurisdictions of less than 25,000 people and those from the northeast.

This report on the patterns and predictors of A-EBPs in health departments provides information on gaps and areas for improvement that can be linked with ongoing quality improvement processes. These activities include practice-based research networks,¹⁰³ public health accreditation efforts,⁶ and several practice-based training programs.^{104, 105} This type of practice-oriented research is promising because it marries university-based inquiry with the real world experience of practitioners.