Barriers to Evidence-Based Decision Making in Public Health: A National Survey of Chronic Disease Practitioners

SYNOPSIS

Objective

Existing knowledge of evidence-based chronic disease prevention is not systematically disseminated or applied. This study investigated state and territorial chronic disease practitioners' self-reported barriers to evidence-based decision making (EBDM).

Methods

In a nationwide survey, participants indicated the extent to which they agreed with statements reflecting four personal and five organizational barriers to EBDM. Responses were measured on a Likert scale from 0 to 10, with higher scores indicating a larger barrier to EBDM. We analyzed mean levels of barriers and calculated adjusted odds ratios for barriers that were considered modifiable through interventions.

Results

Overall, survey participants (n=447) reported higher scores for organizational barriers than for personal barriers. The largest reported barriers to EBDM were lack of incentives/rewards, inadequate funding, a perception of state legislators not supporting evidence-based interventions and policies, and feeling the need to be an expert on many issues. In adjusted models, women were more likely to report a lack of skills in developing evidence-based programs and in communicating with policy makers. Participants with a bachelor's degree as their highest degree were more likely than those with public health master's degrees to report lacking skills in developing evidence-based programs. Men, specialists, and individuals with doctoral degrees were all more likely to feel the need to be an expert on many issues to effectively make evidence-based decisions.

Conclusions

Approaches must be developed to address organizational barriers to EBDM. Focused skills development is needed to address personal barriers, particularly for chronic disease practitioners without graduate-level training.

Seven of every 10 Americans who die each year, or more than 1.7 million people, do so as a result of a chronic disease.¹ In the United States, the direct and indirect costs of the most common chronic diseases totaled $1.3 trillion in 2003, with projected costs expected to reach $4.2 trillion in 2023.² Evidence-based interventions have great potential to impact chronic disease, based on their reliance upon research-proven methods.^3–6

Based on more than a decade of discourse, evidence-based public health is summarized as “making decisions based on the best available scientific evidence, using data and information systems systematically, applying program-planning frameworks, engaging the community in decision making, conducting sound evaluation, and disseminating what is learned.”⁷ Ideally, public health practitioners would make evidence-based decisions by employing these concepts in all chronic disease prevention programs. However, existing knowledge on effective chronic disease prevention is not systematically disseminated or applied.

There are now analytic tools that can identify evidence-based interventions and foster uptake. For example, the Task Force on Community Preventive Services has produced the Guide to Community Preventive Services (hereafter, the Community Guide), enumerating interventions that are effective in preventing chronic diseases.⁵ The topics covered in the Community Guide are diverse and include tobacco, physical activity, nutrition, cancer screening, diabetes, obesity, and other related topics. Despite the availability of such tools, numerous studies show that implementation of evidence-based interventions is low in many clinical and community settings.^8–12

Effective programs do not achieve their potential if they are not disseminated beyond their original testing in research trials.^13–15 Sparse knowledge exists regarding effective approaches for dissemination of research-tested interventions among real-world practice audiences. A systematic review of 35 dissemination studies found no strong evidence to recommend any particular dissemination strategy as effective in promoting the uptake of evidence-based chronic disease control interventions.¹⁶ Further research is necessary to understand the determinants and approaches that will enhance the dissemination of effective interventions. Given the 50 states' constitutional authority to protect the public's health, practitioners in state health departments are in a unique position to implement programs and services related to chronic disease control.^17,18 These practitioners can provide rich insight into the processes by which evidence-based programs are implemented and disseminated.

Both individual and organizational factors can impede public health practitioners' ability to implement evidence-based programs.¹⁹ Previously identified barriers to implementation include lack of skilled personnel, lack of time to gather evidence, inadequate resources and funding, fragmented local and state public health services, and limited buy-in from leadership.^{8, 20–22} Additionally, public health decisions are made on short timescales due to short-term targets and budget cycles, impeding the ability to make longer-term plans that are often necessary for evidence-based interventions.²³

To date, no U.S. studies have identified the salience of barriers to evidence-based practice among a representative sample of chronic disease practitioners. This article explores state and territorial chronic disease practitioners' barriers to evidence-based decision making (EBDM) and how these differ by characteristics of the practitioners. This analysis is part of an ongoing study that aims to increase the dissemination of evidence-based chronic disease interventions in public health agency settings.

METHODS

Survey development

State-level chronic disease practitioners from the 50 U.S. states and territories completed a 74-question online survey from June through August 2008. Along with questions about barriers to EBDM, the survey included questions about the use of the Community Guide and other resources, the importance and availability of key components of EBDM, personal chronic disease-related health behaviors of the respondent, and additional demographic information. Open-ended questions captured qualitative data on participants' perceptions of barriers to EBDM, changes needed to increase EBDM, and other resources for using evidence-based interventions. The survey was designed to be completed in 15 minutes.

The research team developed survey questions based on previous work²⁴ and input from chronic disease practitioners. The survey was tested with a panel of consultants comprising chronic disease experts and a representative sample of the target population of the survey (n=12). After reviewing a draft of the survey, panel members gave cognitive response feedback in a one-hour telephone interview based on methods previously published.^25–28 The research team incorporated this feedback into the final version of the survey.

Because it represents chronic disease practitioners at the state and territorial levels, the membership of the National Association of Chronic Disease Directors (NACDD) was chosen as the target population for this study. All members of NACDD were contacted by e-mail to explain the purpose of the study and the survey. One week later, participants received a link to the survey. Trained research staff phoned each participant to ensure receipt of the survey and to discuss any questions. Reminder e-mails were sent every few weeks until the survey closed.

Of particular relevance to this article were nine statements relating to barriers to evidence-based practices, presented on a Likert scale from 0 to 10, with 0 indicating “strongly disagree” and 10 indicating “strongly agree.” Participants reported the extent to which they agreed with statements reflecting four personal and five organizational barriers. Seven of the nine statements were reverse-coded so that for all barriers, a higher score indicated a larger barrier to EBDM.

Analysis

Of 469 survey respondents (response rate = 65%), 95% reported working in state health departments. Analysis was restricted to those individuals, resulting in a final sample size of 447 participants. Descriptive statistics were calculated for each barrier. Sample sizes varied due to missing data. For barriers with greater than 10% missing data, a Chi-square analysis was performed to compare participants with and without missing data across the variables listed in Table 1. Additionally, the mean score for each barrier was calculated for each U.S. state, excluding states with fewer than four respondents for a given barrier statement. These mean scores were divided into quartiles and mapped to discern geographic patterns. Due to a low number of respondents per state (mean = 8.0, 95% confidence interval [CI] 6.9, 9.0; range: 1–18), state-level results are not reported. However, patterns are presented to inform future research.

Table 1.

Characteristics of state and territorial chronic disease practitioners (n=447) who responded to a national survey on evidence-based decision making, 2008

^aGeneralists included program managers/administrators/coordinators, program planners, division or bureau heads, division deputy directors, department heads, and an academic educator.

^bSpecialists were health educators, epidemiologists, statisticians, program evaluators, community health nurses, social workers, dietitians, or nutritionists.

^cDoctor of Philosophy, Doctor of Public Health, Doctor of Medicine, Doctor of Science, or Doctor of Osteopathic Medicine

^dMaster of Public Health or Master of Science in Public Health

^eBachelor of Arts or Bachelor of Science

In the survey, participants recorded all of the academic degrees that they held. To incorporate this information into multivariate analyses, the research team created a variable for the highest degree achieved. Survey participants were coded in the following order: (1) doctoral degree (Doctor of Philosophy, Doctor of Public Health, Doctor of Science, Doctor of Medicine, or Doctor of Osteopathic Medicine); (2) public health master's degree (Master of Public Health or Master of Science in Public Health); (3) other master's degree; and then (4) bachelor's degree (Bachelor of Arts or Bachelor of Science).

As in previous research, participants selected the job description that best reflected their primary position, and these responses were categorized by the research team into specialists and generalists.¹¹ Generalists were program managers/administrators/coordinators, program planners, division or bureau heads, division deputy directors, department heads, or academic educators. Specialists were health educators, epidemiologists, statisticians, program evaluators, community health nurses, social workers, dietitians, or nutritionists.

The research team determined that lack of skills to develop evidence-based chronic disease programs (“develop”), lack of skills to effectively communicate evidence-based strategies to state-level policy makers (“communicate”), and feeling the need to be an expert on many issues to effectively make evidence-based decisions (“expert”) were the most modifiable barriers. Programs and interventions can be developed to modify these skills and perceptions. For those modifiable barriers, the scores were ranked and placed into tertiles. Using logistic regression models, we calculated adjusted odds ratios (AORs) and 95% CIs to compare those who reported the lowest third of barrier scores with those who reported the highest third. As there was a lack of previous research detailing the variables that impact barriers to EBDM, the following variables were included in a forced-entry, exploratory regression analysis: gender, age, highest degree, primary job, and census region.

Other barriers addressed in this study included fears about personal job security (“job security”), chronic disease prevention not being a high organizational priority (“prevention priority”), lack of organizational incentives/rewards for EBDM (“incentives”), the organizational culture not supporting creative thinking or the use of new ideas (“supportive culture”), inadequate agency funding for developing and implementing evidence-based programs (“funding”), and state legislators being unsupportive of evidence-based interventions and policies (“legislators”). The research team considered expert, communicate, develop, and job security to be personal barriers, while incentives, funding, legislators, supportive culture, and prevention priority were deemed to be organizational barriers.

RESULTS

Table 1 shows selected characteristics of the sample. Among survey respondents who reported working in state health departments (n=447), 80% were classified as generalists and 63% identified their primary position as a program manager/administrator/coordinator. The majority (80%) were female. The average length of experience in public health was 15.4 years (95% CI 14.6, 16.2). The final sample represented all 50 U.S. states, the District of Columbia, American Samoa, Puerto Rico, the Republic of Marshall Islands, and the U.S. Virgin Islands. Delaware, Georgia, Mississippi, Nevada, the District of Columbia, and each of the U.S. territories had fewer than four respondents.

Seventy-one percent (n=318) of participants reported that they were aware of barriers to using evidence-based interventions or making evidence-based decisions. In response to the nine Likert scale barrier statements, higher scores were reported overall for organizational barriers (mean = 5.7; 95% CI 5.5, 5.9) than for personal barriers (mean = 3.6; 95% CI 3.5, 3.7) (Table 2). The most highly endorsed organizational barriers were incentives (mean = 7.7; 95% CI 7.5, 8.0), funding (mean = 7.3; 95% CI 7.0, 7.5), and legislators (mean = 6.4; 95% CI 6.2, 6.7). Among the personal barriers, expert was the major barrier (mean = 6.5; 95% CI 6.3, 6.8). Incentives and legislators each had greater than 10% missing data; however, looking across variables listed in Table 1, participants who did not answer these questions did not significantly differ from those who did (p≤0.05).

Table 2.

State and territorial chronic disease practitioners' self-reported barriers^a to using evidence-based decision making to prevent chronic disease, 2008

^aBarrier scores were measured on a 0–10 Likert scale, with higher scores indicating a greater perceived barrier.

CI = confidence interval

In the exploratory analysis of regional differences in barriers, patterns emerged in several of the maps showing barrier quartiles by state (data not shown). Regarding the develop barrier, a cluster of higher scores (third and fourth quartiles) emerged in the middle of the country. The majority of the highest barriers to legislators stretched across the South, with the third quartile clustering in the states surrounding Ohio. A visible clustering pattern appeared in the funding barrier, with many southern states reporting the highest barriers and states in the Rocky Mountain West reporting the lowest barriers.

When the modifiable barriers were divided into tertiles, the lowest scores corresponded to responses of 0–1 for develop and communicate and 0–5 for expert. The highest scores corresponded to responses of 4–8 for develop, 4–10 for communicate, and 8–10 for expert. Table 3 shows the odds of reporting the highest levels for each of these barriers as compared with the lowest levels.

Table 3.

Odds of reporting highest tertile of scores^a for modifiable barriers to evidence-based decision making in a survey of state and territorial chronic disease practitioners, 2008

^aBarrier scores were measured on a 0–10 Likert scale, with higher scores indicating a greater perceived barrier.

^bAOR compared highest to lowest tertiles of barrier scores; logistic regression model included gender, age, degree, primary job, and census region as covariates.

^cStatistically significant at p<0.05

^dMaster of Public Health or Master of Science in Public Health

^eDoctor of Philosophy, Doctor of Public Health, Doctor of Medicine, Doctor of Science, or Doctor of Osteopathic Medicine

^fBachelor of Arts or Bachelor of Science

AOR = adjusted odds ratio

CI = confidence interval

Men and women reported significantly different barriers in develop, communicate, and expert. Controlling for age, highest degree, primary job, and census region, men were 60% less likely than women to report scores in the highest tertile for the barriers develop and communicate (for both barriers: AOR=0.4, 95% CI 0.2, 0.9). However, men were 2.3 times more likely to report scores in the highest tertile for the expert barrier (AOR=2.3, 95% CI 1.2, 4.6).

Compared with participants with a public health master's degree as their highest degree achieved, those with a bachelor's degree as their highest degree were 5.6 times more likely to report the highest barriers in develop in the adjusted model (AOR=5.6, 95% CI 1.7, 17.9). Significant differences were also found among highest degree and primary job type within the expert barrier. Participants with a doctoral degree, as compared with those with a public health master's degree, were 3.1 times as likely to report the highest barriers to expert, controlling for the other covariates (AOR=3.1, 95% CI 1.2, 8.0). Specialists also were 2.4 times as likely to report the highest expert barriers compared with generalists (AOR=2.4, 95% CI 1.0, 5.4).

Considering census regions, chronic disease practitioners in the South were less likely to report the highest scores for barriers in develop (AOR50.2, 95% CI 0.1, 0.6) and communicate (AOR50.3, 95% CI 0.1, 0.7) compared with the Midwest and more likely to report barriers in expert (AOR52.2, 95% CI 1.1, 4.6). Practitioners in the West were also less likely to report barriers in develop compared with the Midwest (AOR50.4, 95% CI 0.1, 1.0).

DISCUSSION

While experts agree that public health decisions should be science-based, chronic disease control programs are too often developed and implemented without utilizing practices found through research to be effective.⁷ A better understanding of the barriers to the dissemination of evidence-based practices has been needed, particularly in regard to organizational infrastructure barriers.²⁹ While previous work has identified several barriers to EBDM,^{8, 20–22} this analysis was designed to quantify state-level chronic disease practitioners' assessments of how these barriers impact their work.

Participants in this survey identified that organizational factors were more likely to create additional barriers to EBDM than were personal skills and qualities as public health practitioners. Notably, they stated that inadequate funding for evidence-based programs, state legislators who did not support evidence-based interventions and policies, and a lack of incentives or rewards for using EBDM were the predominant organizational barriers to EBDM. While organizational barriers are difficult to modify through interventions, the results of this survey help communicate practitioners' opinions on the barriers that most restrict their practice of EBDM. A Designing for Dissemination conference identified that scientists, practitioners, and policy makers are disconnected in their views of dissemination.³⁰ A growing body of evidence that addresses these disconnections may foster dialogue among the different groups, potentially resulting in funding, legislation, research, and organizational infrastructures that are more supportive of the public's health.

Despite the finding that the personal barriers were not the highest reported barriers, they may be the easiest to target for improvement. EBDM courses, such as the one developed by members of this team,^11,31 could be employed to improve the skills of practitioners. According to the results of the logistic regression analysis in this survey, these courses could be tailored to help improve either the skills or beliefs of those groups that were found to have significant differences in their reports of personal barriers. For example, practitioners with a bachelor's degree as their highest degree were more than five times as likely to report a lack of skills in developing evidence-based programs compared with those who had a public health master's degree. Training courses could be developed to target practitioners without graduate-level training. Specialists (e.g., epidemiologists and health educators) and doctoral-level practitioners may benefit from basic skills training that complements their existing expertise.

Open-ended questions within this survey provided evidence that practitioners independently identified the same barriers that the quantitative aspects of this survey investigated. Participants stated that individuals' skills can impede EBDM, that funding is often inadequate to conduct evidence-based interventions, that political barriers exist, and that organizational barriers (e.g., lack of understanding, flexibility, or support) can undermine EBDM.³² Other qualitative work conducted by members of this team has highlighted the importance of leadership support, consistency in leadership direction, public health training for all staff members, funding, and the turnover of state government and its consequential changes in priorities.³³

Limitations

A major limitation of this study was that data were self-reported. In particular, it is difficult to ascertain the difference between people's report of their personal skills and their actual skill level. One way to address this limitation would be to more objectively assess skills in practice settings (e.g., a manager's assessment) or to survey other stakeholders (e.g., policy makers or agency heads) on their commitment to EBDM. Additionally, those who responded to the survey may have had differing opinions on barriers to EBDM than those who did not respond, although studies of household surveys suggest that nonresponse bias is less likely with response rates higher than 65%.³⁴ Data were not available to compare Table 1 characteristics with the entire NACDD membership.

Future research is needed to determine effective methods of overcoming these barriers (e.g., incorporating EBDM into funding requirements, making EBDM a component of practitioners' performance reviews, and offering tailored training to different types of practitioners). The discovery of geographic patterns in several of the barriers also warrants additional research, including state-specific data collection. States often differ in how they fund programs or in their laws and policies that influence public health spending and practice. Further evidence of regional differences in barriers to EBDM may indicate a need to prioritize funding or other resources by region. Also, this study did not consider the varying organizational structures of state health departments or the political climates within which they work. An important link exists between the structure of a state health department and its functioning capabilities.³⁵

CONCLUSIONS

This survey examined public health practitioners' perceptions of barriers to EBDM. The results may improve state health departments' abilities to facilitate and encourage EBDM and, in turn, assist chronic disease practitioners in implementing chronic disease interventions that have been proven effective. Use of such interventions will improve the health of the public through the prevention of chronic diseases.