Capacity Building Toward Evidence-Based Medicine Among Healthcare Professionals at the University of Medicine and Pharmacy, Ho Chi Minh City, and Its Related Institutes

Abstract

Research capacity development enhances a country’s ownership of activities aimed at strengthening its health system. In Vietnam, continuing medical education (CME) is attracting increasing attention with the establishment of legal and policy frameworks. During 2010-2013, the Japan International Cooperation Agency funded a research capacity building project targeting physicians in Ho Chi Minh City. The project had been developed in four previous courses that were conducted in collaboration with Fukushima Medical University and Ho Chi Minh City University of Medicine and Pharmacy (UMP). The project succeeded in obtaining accreditation as the city’s CME course. A total of 262 physicians attended three courses that have a divided set of research competencies. Following the Kirkpatrick Model for evaluating the effectiveness of training programs, we confirmed the participants’ positive reaction to the courses (Level 1 evaluation), their perceived increase in knowledge and confidence in research skills (Level 2 evaluation), and application of learned knowledge in their practice (Level 3 evaluation). Presented here is a step-by-step scaling-up model of health research capacity building. Strategies for the further expansion include: further capacity building of instructors; responding to clinicians’ specific needs; building a recruiting system with authorization; and improving the Level 3 training evaluation.

Introduction

Health research is a driving force for improving the performance of health systems, defined by the World Health Report 2000 as all activities whose primary purpose is to promote, restore, or maintain health. In the 2008 Toyako Summit and related follow-up activities, health information was emphasized as an important component for strengthening health systems.¹ Research can help countries identify their own needs and communicate research findings toward implementation. More recently, the Inter Academy Medical Panel released a statement, “A Call for Action to Strengthen Health Research Capacity in Low and Middle Income Countries,” which was endorsed in 2013 by more than 40 national academies of medicine worldwide. This is one of the topics discussed in the World Health Summit 2013, held on October 20-22, 2013. To indigenize health research systems in developing countries, it is essential to build health research capacity.

In Vietnam, continuing medical education (CME) has gained increasing attention through the establishment of legal and policy frameworks such as the Law on Medical Examination and Treatment and Circular 07/2008/TT-BYT for continuous training. One of the five policies stated in the law is to encourage scientific and technological research and application to medical examination and treatment; the Ministry of Health is assigned to provide human resource training in this aspect. The circular was issued by the ministry to guide CME, regulating the training curriculum and materials. However, the actual carrying out of these regulations has faced difficulties related in part to the mechanisms of implementation.²

Having recognized the need for institutional capacity development in research, Ho Chi Minh City University of Medicine and Pharmacy (UMP) gained assistance from the Population Council in 2000 to initiate a 9-month program for part-time research training for its staff and clinicians³; from 2004 to 2009, UMP collaborated with the Department of Public Health at Fukushima Medical University (FMU) to conduct four intensive, short-term, full-time training courses for clinicians.⁴ Building upon this experience and to establish a wider scheme for research capacity development, the Japanese International Cooperation Agency (JICA) funded the project “Capacity building toward evidence-based medicine among healthcare professionals at UMP and its related institutes” during the period 2010-2013. The UMP and FMU collaborated to implement the project, with support from the Ho Chi Minh City Medical Association and the Japanese Epidemiological Association. This paper presents results from the final project evaluation and recommendations for institutional research capacity building for physicians in Vietnam.

Methods

Training course

After a training program for trainers was conducted with nine Vietnamese physicians in Japan in 2010, three series of research training courses were implemented at UMP in 2011 and 2012. Each series was 4 to 5 days in duration, and was facilitated by Japanese and Vietnamese lecturers and Vietnamese assistant instructors who had completed the program in Japan. The training curriculum is outlined in Table 1. Course 1 focused on epidemiology, Course 2 on biostatistics, and Course 3 on publication skills. The integration of lectures with hands-on training that included critical appraisal, protocol development, data analysis, and article writing allowed participants to acquire the technical competencies required for research.

Table 1.

Curriculum of the research training courses at the University of Medicine and Pharmacy, Ho Chi Minh City

	Course 1	Course 2	Course 3
Course duration	5 working days (August 15-19, 2011)	4 working days (February 12-15, 2012)	4 working days (July 29-August 1, 2012)
Main objectives	・Reinforce basic knowledge of epidemiology ・Provide technical competencies required for research, particularly critical appraisal and study designing	・Reinforce basic knowledge of biostatistics ・Provide technical competencies required for research, particularly data handling and analysis	・Review basic knowledge of epidemiology and biostatistics ・Provide technical competencies required for research, particularly publication skills
Group tasks	・Article critique ・Development of research proposal	・Analysis of data	・Writing an abstract

Participants

A total of 260 physicians from Ho Chi Minh City and the provinces in the south of Vietnam participated. They could choose to register for either the “audit course” or the “project course.” While the audit course participants only attended lectures and in-class exercises, project course participants were divided into six groups and received support from the instructors’ team to conduct research projects at their hospitals. The project course participants were encouraged to attend three series.

Evaluation

Based on the Kirkpatrick Model for evaluating the effectiveness of training programs,⁵ the evaluation was conducted to assess participants’ reactions to the courses (Level 1 evaluation); participants’ changes in knowledge, skills, attitude (Level 2 evaluation); and behaviors in research and evidence-based medicine (Level 3 evaluation). In addition, ownership and project management were also assessed.

Main data collection methods included course evaluation survey, e-mail survey, semi-structured in-depth interviews, and review of project documents. The course evaluation survey was conducted at the end of each course, asking participants to rate their progress toward training objectives, course usefulness, topic selection, quality of course materials, content level, course length, and their interest in attending similar course in future. Answer options were given on a 5-level scale, with 1 being poor, easy, or short and 5 being superior, difficult, or long depending on the context. The e-mail survey was sent out about 3 months after Course 3 to 160 participants whose e-mail addresses were on record. This follow-up survey asked whether the participants used the skills they had learned: whether they had searched the scientific evidence, analyzed patient data, participated in research, and presented and published research findings. The in-depth interview was conducted among eight participants by the first author (LTQG), asking about the usefulness and relevance of the training, motivators and barriers for applying evidence-based medicine, and suggestions for future courses. The review of project documents focused on project expansion processes, ownership, and management.

Analysis of quantitative data from the course evaluation surveys and an e-mail survey was carried out using SPSS version 10.0.5. In addition to descriptive analysis of survey items, correlations between the number of courses that had been attended and self-evaluated achievements were analyzed.

Ethical consideration

This evaluation of training courses for quality improvement was conducted as a part of the JICA project. All data were collected anonymously without any identification information of participants, and in accordance with Ethical Guidelines for Epidemiological Research issued by the Japanese Ministry of Education, Culture, Sports, Science and Technology and the Japanese Ministry of Health, Labour and Welfare.

Results

A total of 262 physicians participated in three series. The participation rate (number of graduates/registrants) was 64% (100/157) in Course 1, 75% (91/121) in Course 2, and 99% (71/72) in Course 3. The number of participants from the provinces, who received financial support for travel, was nine in Courses 1 and 2 and seven in Course 3. Among participants who responded to course evaluation surveys, the proportion of returning participants, who had attended previous course(s), was 47% (27/57) in Course 2 and 48% (21/44) in Course 3. When courses taken before receiving JICA funding were included, the proportion was 11% (8/71) in Course 1, 54% (31/57) in Course 2, and 50% (22/44) in Course 3.

Level 1 evaluation

Data of key indicators of Level 1 evaluation are summarized in Table 2. Over 80% of participants in three courses evaluated the usefulness of the training contents as superior, with the lowest proportion in Course 2 (84%). Correspondingly, the proportion of participants who answered that the overall level was difficult was highest in Course 2 (75%). Over 85% of the participants in the three courses showed interest in attending similar courses in the future.

Table 2.

Level 1 evaluation

Indicators	Participants’ course evaluation [N (%) of 4 or 5a]
	Course 1 N=70	Course 2 N=57	Course 3 N=44
Usefulness of training contents (superior)	66 (94%)	48 (84%)	40 (91%)
Selection of training topics (superior)	58 (83%)	45 (79%)	34 (77%)
Course materials (superior)	60 (86%)	43 (75%)	37 (84%)
Course duration (long)	22 (31%)	18 (32%)	14 (32%)
Overall level (difficult)	37 (53%)	43 (75%)	22 (50%)
Interest in similar course (superior)	59 (97%)	51 (89%)	38 (86%)

^aA five-level scale was used, with 1 being poor, easy, or short and 5 being superior, difficult, or long depending on context. Proportions of those who answered 4 or 5 in direction indicated in parenthesis are shown in the table.

When interviewed, participants liked practicality of the courses and found them relevant to their work. However, Course 2 was evaluated as difficult, Course 3 attracted fewer participants, and many participants demanded better announcement of the course.

“Most are hands-on skills and steps so they’re easier to apply than theory-based facilitation methods ... I wish I could have attended the previous courses. The master course that I am attending mainly gives lectures on research methods. It is more theoretical while these courses are more practical.”

“The research projects in combination with lectures are helpful for participants to reflect and apply the learned knowledge and skills in real work.”

“There was too much content in the previous training course (Course 2). The lectures were so fast. A lot of content was provided in a training session.”

“Writing abstract (Course 3) seems a long way off as we need to be able to conduct research rigorously by ourselves first.”

“I knew of the course through a friend and did not know about the previous courses. Information about the courses should be disseminated more widely so that more clinicians know. Many clinicians like this type of training course but they do not receive information about it.”

Level 2 evaluation

Table 3 shows the results of the Level 2 evaluation. The proportion of participants who gained knowledge in epidemiology (89% in both Courses 1 and 3) was higher than those who gained knowledge in biostatistics (61% in Course 2 and 66% in Course 3). The percentage for those who gained confidence in conducting research was 64% in Course 1, 44% in Course 2, and 70% in Course 3. Other indicators of confidence in designing research and understanding scientific evidence were also lowest in Course 2. Table 4 shows that the number of past attended courses (since 2004) correlated with the agreement level of gaining confidence in conducting research both in Courses 2 and 3.

Table 3.

Level 2 evaluation

Indicators	Participants’ course evaluation [N (%) of 4 or 5a]
	Course 1 N=64	Course 2 N=57	Course 3 N=44
Increased knowledge of epidemiology	57 (89%)	—	39 (89%)
Increased knowledge of biostatistics	—	35 (61%)	29 (66%)
Gained confidence in conducting research	41 (64%)	25 (44%)	31 (70%)
Gained confidence in designing research	45 (70%)	31 (54%)	32 (73%)
Gained confidence in understanding scientific evidence/articles	54 (84%)	30 (53%)	40 (91%)
Gained confidence in analyzing data	—	25 (44%)	26 (59%)
Gained confidence in writing a scientific paper	—	—	33 (75%)

^a Five-level Likert scale ranging from strongly disagree to strongly agree was used. Proportions of those who answered 4 or 5 in direction of agree are indicated in the table.

Table 4.

Correlation between number of past attended courses and self-evaluated level of confidence

Level 2 indicatorsa	Correlation with the number of past attended coursesb [rs (P value)]
	Course 2	Course 3
Gained confidence in conducting research	0.32 (0.02)*	0.34 (0.03)*
Gained confidence in designing research	0.20 (0.15)	0.21 (0.17)
Gained confidence in understanding scientific evidence/articles	0.38 (0.01)*	0.03 (0.85)
Gained confidence in analyzing data	0.21 (0.14)	0.22 (0.16)
Gained confidence in writing a scientific paper	—	0.24 (0.12)

^a Five-level Likert scale ranging from strongly disagree to strongly agree was used.

^b Total number of courses taken since 2004. In Course 1, the number of past attended courses and the confidence levels were asked on separate sheets and data were not linked.

* Spearman’s correlation is significant at the 0.05 level.

Level 3 evaluation

Among 160 participants with e-mail addresses, 31 (19%) responded to the e-mail survey. Twenty-six (84%) had searched for scientific evidence, 18 (72%) participated in research, 22 (79%) analyzed patient data, 14 (48%) presented, and 11 (38%) published research findings.

All interviewed participants had a plan to conduct research or improve their current research protocols after the course. A few participants planned to share their learned knowledge and skills with colleagues. A highlighted success story of the project is of a returning participant from a province, who attended most of the research training courses since 2004, and who reported increased research capacity, a gain in confidence, and his influence and contribution to strengthening research activities in the health sector in his province in his role as a member of a provincial scientific review committee.

Various institutional and individual barriers existed for applying evidence-based medicine. The barrier that was considered the worst, but if reversed would be the best motivator, is organization leaders’ support.

“It is necessary to change the minds of leaders in the local areas. Sometimes they encourage staff to conduct research just to gain points in the annual assessment checklist. They do not care about the research findings. Therefore, they do not see the value in doing research. It is necessary to have a catalyst.”

“[To motivate research activities] Encouragement from leaders to improve the environment for clinicians can include: financial support for data collection; providing technical assistance from experienced researchers by placing an epidemiologist at every hospital; instituting a method to rate performance ...”

Project ownership and management

The scale of our training expanded, as compared with previous courses before 2010. The very first long-term course funded by the Population Council in 2000 was accredited by the university and targeted only physicians of one department of UMP.³ Subsequent short-term courses were accredited by two universities (UMP and FMU) and targeted physicians from various departments at UMP and UMP-affiliated hospitals.⁴ Currently offered JICA-funded courses have been expanded further; the course was accredited as a Ho Chi Minh City CME course with support from the Ho Chi Minh City Medical Association, and targeted a wider range of clinical practitioners working at hospitals in the city. The course also gained support from the Japan Epidemiological Association.

As mentioned above, the trainers were trained prior to Course 1 in order to establish a local trainer team, consisting of doctors from UMP and the Ho Chi Minh City Medical Association, with the goal of improving their course management and teaching skills. After the training, the local team was involved in developing a syllabus, providing comments on training materials, handling participation recruitment, and the team participated actively in teaching. In Course 1, the main roles of local instructors were to translate the English lectures of Japanese instructors into Vietnamese and facilitate group work. In Courses 2 and 3, one day was set aside to be managed and taught independently by local instructors without the presence of the Japanese instructors. In group research projects, local instructors led the group work, which served as another training opportunity for them to improve their research and teaching skills. Local instructors, in pairs with Japanese instructors, continued to work on publication of their group project findings.

For further upgrading of the project, we consulted the Hanoi School of Public Health, and obtained advice to reach out more to physicians from the provinces by using existing government recruitment flows in order to fill in the regional gaps in training. The school offered continuous support to the project to develop a course textbook, so the course could be accredited at the national level.

Discussion

The course has been constantly upgraded through building partnerships with local authorities. Both quantitative and qualitative data show that the participants had positive reactions to the training courses (Level 1), showed good learning results—especially in epidemiology (Level 2), and took part in research activities after finishing the courses (Level 3). The significant correlation between the number of past attended courses and the level of gain in confidence in research may reflect the positive effects of the training we offered.

There were four major challenges we faced. First, Course 2 on biostatistics presented lower course evaluation and participants rating of their confidence gained in multiple competencies. This could be explained in part by the level of difficulty of this course as perceived by the participants. Second, Course 3 on publication skills attracted fewer numbers of participants because the participants needed more basic support to plan and start their research, and local instructors themselves needed more training in this aspect. Third, many participants showed interest in attending similar courses and requested wider recruitment and broader announcement of the course. There were only a total of 25 participants from the provinces in the present courses. Fourth, the response rate of the e-mail survey to confirm the application of learned competencies was too low to conduct a rigorous Level 3 evaluation. Even among the respondents, who were assumed to be highly motivated participants, a smaller proportion of them presented and published their research findings than the proportion participating in research activities.

For future courses, four strategies are recommended, following advice from Hanoi School of Public Health. The first strategy is to increase the capacity building of the instructors’ teaching skills. Also, a review of the methods used to teach practical biostatistics knowledge and data analysis competencies is suggested, as well as more experience in publication among local instructors. The second strategy is to match participants’ needs with the teaching content. Teaching clinical practitioners how to publish articles in academic journals should be reconsidered, and this part of the course might better target those who are in teaching positions. The third strategy is a suggested expansion of target areas. With support from the Hanoi School of Public Health and also obtaining accreditation from the Ministry of Health, the project will be able to take advantage of the existing government recruitment flow. The Circular 07/2008/TT-BYT requires accreditation from the ministry to institutionalize region-wide CME activities. The fourth strategy is to improve project evaluation, which is a necessary condition for ongoing improvements. Kirkpatrick, whose evaluation model we applied, recently stated that many training professionals stay at Kirkpatrick’s first two evaluation levels.⁶ He urges that trainers should “begin from the end”; if we cannot describe how our training positively affects our ultimate goal, our initiative is not really “mission-critical.” Involving leaders of participants’ affiliated hospitals could be the first step toward gaining their understanding of and encouragement of research activities.

Even in developed nations, integration of research capacity building into physician training is still under development. For example, the Accreditation Council of graduate medical education in the United States requires all accredited internal medicine residency training programs to engage residents in research and a trial of research rotation, as reported in 2006.⁷ In Australia, the Research Development Program was initiated in 2004 to promote research among general practitioners.⁸ In Japan, residency programs still strive to secure adequate time for education in general.⁹ Short-term intensive training such as our 4- to 5-day courses and the 5-weekend research capacity building program in Ontario for family physicians¹⁰ could nurture research culture among busy clinical practitioners if carefully planned and well implemented.

Limitations

There are two methodological limitations to this study. First, this study had neither a control group nor a before-after comparison as an intervention study. Second, the obtained results should be carefully interpreted when replicating a similar project, given the possibility of overestimating results due to the potential synergy between enthusiastic organizers and participants. For instance, nearly half the participants in Courses 2 and 3 had attended previous course(s).

Conclusion

Presented here is a scaling-up model of health research capacity building from department, university, and finally to city level. Ownership of local counterparts has been built through partnerships with the Japanese team and other authorizing institutions. Local assistant instructors have been increasingly involved in course management and teaching. Furthermore, analysis and publication of research achievements before scaling up to the next level serves yet another model of evidence-based medical human resource development.^3,4 Currently, we are aiming to expand our training further to a region-wide project that targets provinces in the South of Vietnam.¹¹

Acknowledgments

We thank all organizers, instructors and assistant instructors for their contributions in implementing the courses; Seiji Yasumura (Fukushima Medical University); Chihaya Koriyama (Kagoshima University); Hirohide Yokokawa (Juntendo University); Yuriko Suzuki (National Institute of Mental Health); Vo Tan Son, Tran Diep Tuan, Pham Qui Trong, Nguyen Thi Tu Van, Tran Quang Nam, Tran Viet Thang, Nguyen Thu Tinh (Ho Chi Minh City University of Medicine and Pharmacy); Vo Tuan Khoa, Nguyen Thi Boi Ngoc (People’s Hospital 115).

This project was supported by the Japan International Cooperation Agency Partnership Program (Local government type).