Building the capacity – examining the impact of evidence-based public health trainings in Europe: a mixed methods approach

Abstract

Objective

Since 2002, a course entitled ‘Evidence-Based Public Health (EBPH): A Course in Noncommunicable Disease (NCD) Prevention’ has been taught annually in Europe as a collaboration between the Prevention Research Center in St Louis and other international organizations. The core purpose of this training is to strengthen the capacity of public health professionals, in order to apply and adapt evidence-based programmes in NCD prevention. The purpose of the present study is to assess the effectiveness of this EBPH course, in order to inform and improve future EBPH trainings.

Methods

A total of 208 individuals participated in the European EBPH course between 2007 and 2016. Of these, 86 (41%) completed an online survey. Outcomes measured include frequency of use of EBPH skills/materials/resources, benefits of using EBPH and barriers to using EBPH. Analysis was performed to see if time since taking the course affected EBPH effectiveness. Participants were then stratified by frequency of EBPH use (low v. high) and asked to participate in in-depth telephone interviews to further examine the long-term impact of the course (n = 11 (6 low use, 5 high use)).

Findings

The most commonly reported benefits among participants included: acquiring knowledge about a new subject (95%), seeing applications for this knowledge in their own work (84%), and becoming a better leader to promote evidence-based decision-making (82%). Additionally, not having enough funding for continued training in EBPH (44%), co-workers not having EBPH training (33%) and not having enough time to implement EBPH approaches (30%) were the most commonly reported barriers to using EBPH. Interviews indicated that work-place and leadership support were important in facilitating the use of EBPH.

Conclusion

Although the EBPH course effectively benefits participants, barriers remain towards widely implementing evidence-based approaches. Reaching and communicating with those in leadership roles may facilitate the growth of EBPH across countries.

Introduction

Evidence-based public health (EBPH) has been concisely defined as ‘the process of integrating science-based interventions with community preferences to improve the health of populations’ (1). Other descriptions appear to come to a consensus that evidence-based decision-making processes should include ‘a combination of scientific evidence, as well as values, resources, and context’ (2). Successfully using these EBPH processes could help alleviate negative health outcomes (e.g. noncommunicable diseases (NCDs)), as well as improve the public’s health (3). However, decisions in the realm of public health are usually based on more short-term opportunities, or hot topics that garner the public’s or other interest group’s attention.

Benefits to using a more evidence-based decision-making process include utilization of higher-quality information on what works, higher likelihood of evidence-based programmes being implemented, greater productivity in the workforce, more efficient use of public health resources, and an overall improvement in public health (3). These benefits of EBPH will not only help in promoting health, but in carrying out the WHO’s Global NCD Action Plan 2013–2020 (hereafter called 2020 Action Plan) (4). The burden of NCDs (e.g. cardiovascular diseases, cancers, chronic respiratory diseases and diabetes) is devastating as more people die from NCDs worldwide than any other cause (5). The 2020 Action Plan requires increasing public health capacities at both national and international levels in order to combat NCDs (4). This will help ensure the implementation of recommended NCD interventions, as well as the evaluation of 25 indicators at the country level.

Still, barriers remain towards the use of EBPH in any decision-making process, including the implementation of the 2020 Action Plan. The most significant barrier that exists is the lack of knowledge of EBPH. Most public health workers have no formal training in any of the public health disciplines (e.g. epidemiology, health promotion) (6). Public health structures reflect this lack of knowledge of evidence-based decision-making. Building the capacity to support and sustain EBPH is crucial towards any successful health promotion efforts (7).

An EBPH course was designed to help strengthen the capacity of public health professionals in Europe and internationally to apply and adapt evidence-based programmes in NCD prevention and health promotion. Since 2002, this course entitled ‘Evidence-Based Public Health (EBPH): A Course in Noncommunicable Disease Prevention’ has been taught annually in Europe as a collaboration between the Prevention Research Center in St Louis, the WHO Regional Office for Europe, the Collaboration for Integrated Noncommunicable Diseases (CINDI) and the Centers for Disease Control and Prevention (CDC). Participants of the course are nominated by NCD experts belonging to CINDI, which covers countries in the WHO European Region. The course involves nine interrelated modules that enhance knowledge and skills including: (a) introduction to key concepts; (b) community assessment; (c) quantifying the issue; (d) developing a concise statement of the issue; (e) searching and summarizing the scientific literature; (f) developing and prioritizing programme and policy options; (g) economic evaluation; (h) developing an action plan and implementing integrated prevention interventions; and (i) evaluating the programme or policy. Greater details on the modules and the course are available elsewhere (8,9).

The present study uses a mixed methods approach to assess (a) the utilization and effectiveness of materials/knowledge/skills gained from the EBPH course, as well as (b) general issues faced by public health professionals when using EBPH. The purpose of this study is to assess the effectiveness of the EBPH course over time, while understanding barriers faced in order to inform and improve future EBPH trainings. Though previous studies have performed similar evaluations (10,11), no study has assessed this over as great a period of time, or combined a qualitative assessment to better understand the quantitative findings.

Methods

The present analysis evaluated public health professionals who participated in ‘Evidence-Based Public Health (EBPH): A Course in Noncommunicable Disease Prevention’, during the past 10 years (2007–2016). In total, 208 past participants were contacted via email and asked to take a brief (15 minute average) survey in Qualtrics (12). For increased response rates, participants received four reminder emails, including a final reminder email. Twenty participants were deemed unreachable due to invalid emails, and no other form of communication to contact them. The final response rate was 45.7% from a possible 188 respondents (86 of a possible 188).

In addition to background characteristics, the online survey included questions on the frequency of use of materials, knowledge and skills from the course, reasons for not using course materials and resources as much as intended, benefits from attending the course, leadership support for using EBPH and perception of the role the course has in building capacity for the 2020 NCD Action Plan. A five-point Likert scale was used for the frequency of use of materials/knowledge/skills (seldom/never, annually, quarterly, monthly and weekly). Reasons for not using the course materials and benefits from attending the course used a five-point Likert scale of strongly disagree to strongly agree. Participants were also asked how supportive leaders were of EBPH in their organizations. Responses ranged from ‘Not at all supportive’ to ‘Extremely Supportive’ on a five-point Likert scale. Finally, they were asked whether or not they felt the course contributed to the capacity of implementing the NCD strategies per the 2020 Action Plan.

To explore participant characteristics and responses, we calculated frequencies and descriptive characteristics. In order to evaluate the effect of the course across time, we compared data across two groups – participants during the years 2013 through 2016 (n = 44), and participants during the years 2007 through 2012 (n = 42). Independent samples t-tests were used with statistical significance set at p < 0.05. All quantitative analysis was conducted using SPSS (13). For the qualitative analysis of the interviews, direct quotes were selected to represent the main categories that emerged. The present study was approved by the Institutional Review Board of [de-identified information]

From the survey, 20 participants agreed to be contacted for in-depth telephone interviews in order to further examine the long-term impact of the course. Participants were stratified by mean frequency of use of materials/skills into ‘low-use’ and ‘high-use’ groups, in order to see if there were any differences in responses with regards to use of EBPH. A trained Research Assistant (RA) conducted six interviews with ‘high-use’ participants, and five interviews with ‘low-use’ participants for a total of 11 interviews. The interviews lasted an average of 27 minutes and included questions to expand on how EBPH should be defined, usefulness of course material/knowledge/skills, value placed on EBPH and barriers toward using EBPH. Interviewees were also asked to remark on the utility of specific modules within the EBPH course.

All interviews were recorded with the participant’s permission and professionally transcribed. Authors created a codebook with anticipated themes that would arise. During the process of coding interviews, authors then used deductive focused coding techniques to better represent themes present in the transcriptions. Themes were summarized for all interviews and then labelled into categories. All transcriptions were coded and summarized in NVIVO 11 (14).

Results

Participant characteristics

Among respondents, the majority came from federa/government/local/regional/provincial health departments (51%), with an additional 26% from university settings and 1% from community-based organizations (Table 1). Around 37% held some form of medical doctor degree (MD, Dr.med, MBBS, or DO), while almost 33% held another form of doctorate (PhD, DrPH, DPhil, or ScD). Over a quarter of respondents held a master’s degree in public health (27%), and 27% held other master’s degrees. Other degrees held included a RN (2%), RD (1%) and a bachelor’s degree (12%). When describing job type, 42% identified themselves as managers, 25% specialists and 23% academics. Participants reported a mean of 11 years working in public health (SD = 8.6 years).

Table 1.

Characteristics of participants in online survey of international evidence-based public health course, by years since course (n = 86 participants), 2007–2016.

Agency	Total	> 4 years since EBPH	⩽ 4 years since EBPH
No. of respondents	84	41	43
Local/Regional/Provincial/City Health Department	15 (17.9)	8 (19.5)	7 (16.3)
Federal or Government Health Department	28 (33.3)	12 (29.3)	16 (37.2)
University	22 (26.2)	11 (26.8)	11 (25.6)
Community-Based Organization	1 (1.2)	-	1 (2.3)
Other	18 (21.4)	10 (24.4)	8 (18.6)
Degrees held (all that apply)
No. of respondents	86	42	44
MD, Dr.med, MBBS, or DO	32 (37.2)	18 (42.9)	14 (31.8)
PhD, DrPH, DPhil, or ScD	28 (32.6)	23 (54.8)	5 (11.4)
MPH or MSPH	23 (26.7)	11 (26.2)	12 (27.3)
MS, MSc, MA, or other Masters	23 (26.7)	8 (19.0)	15 (34.1)
RN	2 (2.3)	1 (2.4)	1 (2.3)
RD	1 (1.2)	1 (2.4)	-
BA or BS	10 (11.6)	3 (7.1)	7 (15.9)
Job Type
No. of respondents	84	41	43
Specialista	21 (25.0)	11 (26.8)	10 (23.3)
Managerb	35 (41.7)	16 (39.0)	19 (44.2)
Academicc	19 (22.6)	8 (19.5)	11 (25.6)
Other	9 (10.7)	6 (14.6)	3 (7.0)
Years in public health, mean (SD)
No. of respondents	82	40	42
	11.0 (8.6)	12.9 (9.3)	9.3 (7.7)

^aSpecialist includes health educator, epidemiologist, statistician, programme planner, programme evaluator and physician.

^bManager includes programme manager, administrator, or coordinator, division or bureau head, division deputy director and department head.

^cAcademic includes academic researcher or academic educator.

Use of course materials, knowledge, and skills

There was variation in the monthly of use of materials, knowledge and skills from the EBPH course (Table 2). The most commonly used skill was searching the scientific literature for information on programmes and interventions (39%), followed by using EBPH materials, resources and skills monthly in planning a new programme/intervention (25%), modifying an existing programme/intervention (25%), referring to the EBPH readings that were provided (22%), and in evaluating a programme/intervention (21%). Few respondents used the EBPH materials, skills and resources monthly to write up the results of a programme/intervention (evaluation) (11%), and for grant applications (6%). There were no significant differences by years since the participant had last taken the EBPH course (> 4 years since v. ⩽ 4 years since).

Table 2.

Percentage of respondents indicating agreement with use of, benefits of and barriers to using international evidence-based public health course materials, by years since course taken (n = 86 participants), 2007–2016.

Survey Item	All Participants, % (na)
	Total (n = 86)	> 4 years since EBPH (n = 42)	⩽ 4 years since EBPH (n = 44)	Pb
On average, every month since the EBPH I have
Referred to the EBPH readings that were provided	22 (19)	17 (7)	27 (12)	0.43
Used the EBPH materials/skills in planning a new programme/intervention	25 (21)	22 (9)	27 (12)	0.46
Used the EBPH materials/skills in modifying an existing programme/intervention	25 (21)	17 (7)	32 (14)	0.16
Used the EBPH materials/skills for grant applications	6 (5)	2 (1)	9 (4)	0.91
Used the EBPH materials/skills in searching the scientific literature for information on programmes/interventions	39 (33)	32 (13)	46 (20)	0.43
Used the EBPH materials/skills in evaluating a programme/intervention	21 (18)	22 (9)	21 (9)	0.58
Used the EBPH materials/skills to write up the results of a programme/intervention	11 (9)	10 (4)	11 (5)	0.65
The EBPH course content helped me to
Acquire knowledge about a new subject	95 (81)	93 (38)	98 (43)	0.58
See applications for this knowledge in my work	84 (71)	80 (33)	86 (38)	0.55
Make scientifically informed decisions at work	81 (69)	83 (34)	80 (35)	0.98
Communicate better with co-workers who use EBPH skills	79 (67)	78 (32)	80 (35)	0.58
Read scientific reports and articles	76 (64)	70 (28)	82 (36)	0.50
Obtain funding for programmes at work	32 (27)	33 (13)	32 (14)	0.49
Develop a rationale for a policy change	65 (55)	61 (25)	68 (30)	0.70
Identify and compare the costs and benefits of a programme or policy	56 (48)	61 (25)	52 (23)	0.50
Prepare reports for policymakers	60 (51)	63 (26)	57 (25)	0.62
Adapt an intervention to a community’s needs while keeping it evidence based	71 (60)	61 (25)	80 (35)	0.18
Teach others how to use/apply the information in the EBPH course	73 (62)	76 (31)	70 (31)	0.92
Become a better leader who promotes evidence-based decision-making	82 (70)	78 (32)	86 (38)	0.53
I have not used the EBPH course content as much as I would like because
I do not have enough time to implement EBPH approaches	30 (25)	35 (14)	25 (11)	0.35
The information was too complex	7 (6)	5 (2)	9 (4)	0.89
The information lacked relevance	4 (3)	3 (1)	5 (2)	0.18
There was too much information and not enough time to process it	7 (6)	10 (4)	5 (2)	0.77
My organization does not have a culture that supports the use of EBPH approaches	25 (21)	28 (11)	23 (10)	0.20
Within my agency, there are no incentives to use EBPH	23 (19)	30 (12)	16 (7)	0.23
There is not enough funding for continued training in EBPH	44 (37)	53 (21)	36 (16)	0.09
The people I work with do not have EBPH training	33 (28)	43 (17)	25 (11)	0.22
Use of Sum Scorec, mean (SD)
No. of respondents	85	41	44	0.48
	1.4 (0.9)	1.4 (0.8)	1.5 (0.9)

^aNot all survey respondents answered all questions. The n’s in parentheses indicate the number of respondents who answered the question. Percentages were calculated by using the number who answered the question.

^bDetermined by independent samples t-test.

^cAveraged responses from use of EBPH section, ranged from 0 ‘Seldom/Never’ to 4 ‘weekly’.

Similarly, during the interviews participants were able to describe EBPH as a process of using evidence (e.g. scientific literature, data, reviews) to inform decision-making processes. Participants understood the importance of using reliable sources for programme planning, while also understanding the need for adaptation of initiatives based on ‘local data’ sound evaluation:

I would say that evidence is not just what the scientific literature says, but then also evaluation of existing programs and local data that can account the local values, basically, the local circumstances … So evidence-based public health is not just thinking of how they’re going to fix the issue, but how do you go about it. So basically going through all the steps of gathering the evidence, talking to your stakeholders, planning the program, and so on and so forth.

Participants discussed instances in their own work where an evidence-based approach was used, and they were able to make sound decisions. One of the major benefits stated by participants was the ability to correctly evaluate a programme, in this case conducting an economic evaluation:

For me, economic evaluation, because actually that’s one of the things that actually I’m doing in my PhD research. I’m actually economically and clinically evaluating different kind of intervention, anti-smoke intervention, among different type of people. So actually for me, economic evaluation was really, a really good model and actually I was really interested in.

Barriers toward using EBPH

When examining the barriers toward using EBPH course content (Table 2), 44% of respondents reported not having enough funding for continued training in EBPH, followed by co-workers not being trained in EBPH (33%), and not having enough time to implement EBPH approaches (30%). A quarter of respondents reported not having the culture that supports the use of EBPH approaches (25%), though 72% of respondents reported having moderately supportive leadership with regards to EBPH use in their organization. Additionally, 23% reported no incentives towards using EBPH. The least reported barriers included that the information is too complex (7%), too much information and not enough time to process it (7%), and that the information lacked relevance (4%). There were no significant differences in reported barriers by years since the participant had last taken the EBPH course, though more recent participants generally reported fewer barriers.

Though survey respondents reported moderately supportive leadership, interview participants who worked in hospital settings and were considered low EBPH use respondents, reported unsupportive leadership as a barrier. Here a participant was discussing value placed on EBPH:

But the rest of the organization, which I mean the administration, the higher administration, or the CEO, doesn’t value the same things which I value. So when you talk about public health and prevention, and the preventive medicine in general, you have to make a team with your colleagues or with the clinicians. And you have to make them understand that if they do prevention, then they can avoid complications in patients and among also the health care workers.

However, most interview participants who worked in more public health-oriented settings and were considered high EBPH use respondents, reported having supportive leadership:

I think that … I think that it’s quite high, because our organization is the research and policy organization. So in all the departments, we are very much like depending on the data for all our policy decisions and shaping the progress.

Mirroring the survey results, interview participants expanded on lack of funding, time, and knowledge as barriers towards using EBPH:

One of course, is the lack of resources, the fact that you don’t have enough people or money to commission say a review of the literature. And so even though you know that that would be the most appropriate first or next step, wherever you are in the process, you don’t do it because of a lack of resources. So this is one of the barriers.

So mainly the barriers as approach, the first is timing, because, sometimes our decisions are not really well thought out because of the immediate decision making. And the other thing is, as I said, like not exactly understanding and not having enough capacity of understanding.

Course benefits

Participants reported numerous benefits from taking the EBPH course (Table 2). The most commonly reported benefits included acquiring knowledge about a new subject (95%), seeing applications for this knowledge in my work (84%), becoming a better leader who promotes evidence-based decision-making (82%), and making scientifically informed decisions at work (81%). The majority of respondents also reported communicating better with co-workers who use EBPH skills (79%), reading scientific reports and articles (76%), teaching others how to use/apply the information in the EBPH course (73%), adapting an intervention to a community’s needs while keeping it evidence-based (71%), developing a rationale for a policy change (65%), preparing reports for policymakers (60%), and identifying and comparing the costs and benefits of a programme or policy (56%). Additionally, 41% of respondents felt that the EBPH course has contributed to the capacity of implementing the NCD strategies per the 2020 Action Plan. There were no significant differences in reported benefits by years since the participant had last taken the EBPH course.

In addition to this, interview participants expanded on the application of this knowledge in their own work:

So it really helps me to not go around the issues, but be a bit more effective in, first, identifying what is the question, then looking for the approaches to the question, which is already known and proved by good evidence, supported by the good evidence, and then, how to translate it into the policy options and how to prioritize according to that.

Discussion

This evaluation provides support for the effectiveness of a course designed to increase skills and knowledge in EBPH among public health professionals in Europe. Though previous studies have shown positive results when examining the use of materials, knowledge, and skills from the EBPH course along with the benefits of the course content (10,11), the present study highlights the effectiveness of the course over time and triangulates quantitative and qualitative methods. A similar study examining differences between this course and a US-based course (10), found almost identical rates of use as this Europe-based EBPH course. However, the use of EBPH was not examined over time. The data presented here are from a sample of participants who took the course during the past 10 years (2007–2016) and provides a long-term perspective on the use of information from the course along with the benefits of this information. The majority of respondents agreed with almost every benefit of the course content, and there was no significant difference over time. However, barriers still remain towards the use of evidence-based decision-making.

The three most commonly reported barriers included not having enough funding for continued training in EBPH, followed by co-workers not being trained in EBPH, and not having enough time to implement EBPH approaches. These barriers also emerged from the interviews, as participants expanded on the lack of funding needed to support evidence-based decision-making, the lack of knowledge within an organization and the lack of time to conduct sound decision-making as important to consider in an environment that is unsupportive of EBPH. Previous studies have also identified these as barriers (10,11), as they are essential towards building capacity for evidence-based decision-making (7). Lack of support from the organization or leadership also emerged as an important barrier in participants who identified as ‘low-use’ EBPH users. However, this could be the result of the work setting. Most participants who shared a lack of support worked in a hospital setting where administration decision-making processes may be more centralized and top-down. In these settings, priority setting processes may be seen as less participatory due to the lack of input coming from middle-level managers and clinicians (15). Furthermore, leadership support represents one of the five domains of capacity building and is essential in promoting EBPH (7).

These barriers suggest the need to expand this EBPH course and create others like it. Building the capacity (e.g. knowledge, funding, support) for the use and sustainability of EBPH is crucial towards future public health efforts, including the implementation of the NCD strategy. Modest investments in building capacity are essential in moving science to practice (7). As several participants felt that this EBPH course has contributed to the implementation of the NCD strategy, it is important to produce public health professionals, including leaders, who are adept in evidence-based decision-making processes. Creating a culture that is supportive of evidence-based decision-making is critical in promoting EBPH (16).

Future courses in EBPH should target those in leadership roles in order to promote changes to the culture of public health work settings. Targeting these decision makers may increase investments in building capacity, which has continuously been highlighted as a necessary step in promoting EBPH. In addition, future courses could involve more distance learning approaches to enhance the reach and efficiency of capacity building.

Limitations

The use of cross-sectional data limits our ability to determine causality. Additionally, the significant time gap since participants had last taken the course (up to 10 years) resulted in a proportion of participants who were unreachable or may have been in a different (non-public health) role since taking the course. There is potential bias in who was interviewed: the sample of participants who agreed to interview may not accurately reflect the range of other participants and we may be missing information from them. Since participants hail from countries across the WHO Europe region, there is unlikely to be homogeneity in education, training and experience. This high variation in prior training is a potential limitation, as it may moderate the impact of training in EBPH. However, this diversity in backgrounds, including this mix of local, regional, and national public health practitioners highlight the overarching impact of this EBPH course. Lastly, increasing knowledge of, and providing resources for EBPH indirectly, rather than directly, impacts population health as there are other factors involved (e.g. politics, ideology, civil society). However, this is an important step in building the capacity to make informed decisions to address public health issues, specifically, NCDs.

Conclusion

‘Evidence-Based Public Health (EBPH): A Course in Noncommunicable Disease Prevention’, is an effective strategy to increase the use of, as well as capacity for evidence-based decision-making internationally. Continuing to build the capacity for EBPH can help alleviate barriers to the use of EBPH, support the implementation of the NCD strategy and address health equity. Support of EBPH, including targeting those in leadership roles, may help facilitate the growth of evidence-based decision-making across different countries. This in turn will lead to improvements in population health and reductions in NCD.