Operational research training: the course and beyond

Abstract

Insufficient operational research (OR) is generated within programmes and health systems in low- and middle-income countries, partly due to limited capacity and skills to undertake and publish OR in peer-reviewed journals. To address this, a three-module course was piloted by the International Union Against Tuberculosis and Lung Disease and Médecins Sans Frontières in 2009–2010, with 12 participants. Five received mentorship and financial support as OR Fellows. Eleven of 12 participants submitted a paper to a peer-reviewed journal within 4 weeks of the end of the course. Evaluation shows that participants continued OR activities beyond the course. During the subsequent year, they submitted and/or published 19 papers, made 10 posters and/or presentations, and many participated in training, mentoring and/or paper reviewing. Some described changes in policy and practice influenced by their research, and changes in their organisation’s approach to OR. They provided recommendations for improving and expanding OR. We conclude that participants can, with certain enabling conditions, take research questions through to publication, use skills gained to undertake and promote OR thereafter and contribute to improvement in policy and practice. An internet-based network will provide participants and graduates with a platform for collection of course outcomes and ongoing mentor- and peer-based support, resources and incentives.

In recent years, operational research (OR) has become an important topic on the international health agenda. Some funding has been made available through mechanisms such as the Global Fund to Fight AIDS, Tuberculosis and Malaria. OR should be an integral part of routine programmes, health systems and services in low- and middle-income countries and, when linked to monitoring and evaluation systems, can lead to improvements in performance at all levels, thus improving patient care and the prevention and management of disease.¹ However, despite the interest in and recognition of its value, relatively little OR is being implemented within programmes and health systems in low-income countries, and even less is being published.

This lack of performance may be due to insufficient capacity and skills to design, undertake, analyse and write up OR to a quality standard adequate enough for publication in peer-reviewed national and international journals. Over the last two decades, there have been many training programmes to build OR capacity. For example, the International Union Against Tuberculosis and Lung Disease (The Union) and the Centers for Disease Control and Prevention, Atlanta, GA, USA, have for many years coordinated and run short training programmes on OR, and large numbers of tuberculosis (TB) programme and laboratory staff have been trained in OR methods. This has produced many protocols, but few published articles.² Another example is the International TB Training Course in Japan. Over a 7-year period between 2001 and 2007, 28 participants developed OR projects, but none led to the publication of a scientific paper.³

New paradigms of training linked to defined outputs are clearly needed. In response, The Union and Médecins Sans Frontières (MSF) have developed a new three-module training course where the success or failure of the course is judged by the proportion of participants who take a research project through from start to finish, submit a paper to an international journal and have it accepted for publication. Although we track the proportion of papers submitted and accepted for publication from each course, we also wish to assess outcomes beyond the course. Post-course outcomes are important measures of success, as the ultimate goals for OR should be the development of research skills and leadership in countries and changes in policy and practice.^4,5 For these outcomes, we need to find out what happens to participants after course completion in terms of their engagement with research in their respective countries and their influence on policy and practice.

The aim of this evaluation was to report on outcomes from an OR training initiative in terms of the research activities undertaken by participants during the year following the end of the course. Specific objectives were to report on 1) research output, 2) research activities undertaken by participants during the year following the end of the course, and 3) qualitative outcomes, including perceived changes to policy, practice and organisational approaches to OR, challenges still faced by participants, and their recommendations for improving and expanding research activities.

METHOD

Setting

The first OR course was run in Paris in 2009–2010 by The Union’s Centre for Operational Research and MSF’s Brussels Operational Centre. The OR course consisted of three modules of 1 week each, with clearly defined outputs for each module.^6–8 Participants who do not achieve these milestones, for whatever reason, are eliminated from the course, and recorded as course failures. Twelve participants were selected according to the criteria shown in Table 1. The 12 participants included five persons who had also been appointed as OR Fellows. Fellows have more stringent selection criteria, must submit two research papers per annum to have their 12-month contracts renewed, receive support to present their work at conferences and opportunities to facilitate in Union/MSF courses and progress to the role of senior Fellow.^6–8 Four of these Fellows received support directly from The Union, and one was supported as part of a memorandum of understanding between The Union and a local institution. The structural details of the course with the three modules are highlighted in Table 2. Briefly, Module 1 was devoted to the development of protocols; Module 2, quality-assured data capture and analysis; and Module 3, writing up the paper.

TABLE 1.

Selection criteria for candidates for The Union/MSF operational research course

Engagement with a national programme and a statement that the participant will return to the programme Completion of a Masters in Public Health or the equivalent or a candidate who is strongly recommended Supervisors’ written endorsement that there is time and opportunity for the participant to carry out research A stated and acceptable mentor at country level Proven competency in English and use of a computer A statement that research funding can be acquired through sources other than the course Submission of a one-page summary of the programme/health system problem and the research question that may be developed into a research protocol, which accompanies the application form

The Union = International Union Against Tuberculosis and Lung Disease; MSF = Médecins Sans Frontières.

TABLE 2.

The Union/MSF course on operational research

Module 1: Protocol development (5 days)

Define the research question Develop the protocol Consider and manage the ethical components of the research Outcome on Day 5: written draft protocol

Between Module 1 and Module 2: refine the protocol and submit for ethical approval to the Union Ethics Advisory Group and the local (country or area) ethics committee

Module 2: Data management and analysis (5 days)

Develop a data entry tool Develop skills for data analysis Outcome on Day 5: draft electronic data entry instrument

Between Module 2 and Module 3: complete the study, enter the data to an electronic software package and analyse the data

Module 3: Paper writing (5 days)

Learn the principles of writing a scientific paper Learn how to deal with on-line submission Learn how to deal with peer review, write point-by-point responses to editors and reviewers and revise the paper Outcome: written draft paper

After Module 3: finalise draft paper and submit to peer-reviewed journal (Module 1 and 2 are 2 months apart, and Module 2 and 3 are 6–7 months apart)

The Union = International Union Against Tuberculosis and Lung Disease; MSF = Médecins Sans Frontières.

Assessment of outputs from participants’ course projects

The success of this first course was judged on a measurable product, namely completed projects written up and submitted to peer-reviewed journals within 4 weeks of the course end. This demonstrated participants’ ability to develop research questions and move through protocol writing, study implementation and data analysis to writing and submission. During the next 24 months we maintained contact with the participants and documented whether these papers were published.

Assessment of post-course outcomes

All 12 participants were included in this assessment, which was undertaken 16 months after the end of the course. A questionnaire was piloted with two of the participants, adjusted and then e-mailed in August 2011 to the other 10 participants. The first section asked about research undertaken in the year after the course, whether the studies had been submitted for publication and published, and whether the participants believed the research had contributed to any changes in policy or practice. The second section asked about post-course involvement in research training, mentoring and paper reviewing, and whether participants had observed any changes in their organisation’s approach to OR following the course. The third section asked participants to rate challenges experienced in implementing OR since the course, and to recommend how OR should be improved and expanded at national, regional and international levels. The questionnaires were stored and analysed in a password-protected file using the qualitative software ATLAS-ti (ATLAS.ti, Berlin, Germany).

Quarterly reports submitted by OR Fellows to The Union as part of their funding requirements were an additional source of information about papers submitted and/or published, and were used to check Fellows’ questionnaire replies. No appropriate, additional source was identified for checking non-Fellows’ replies. Publication details were validated by searches on the PubMed database and journal websites. The status of papers reported as ‘submitted’ in the questionnaire was checked, and updated where necessary, at the time of preparing this manuscript.

Ethics

The Ethics Advisory Group of The Union determined that ethical clearance was not required for this evaluation.

RESULTS

The first operational research course

There were 92 applicants for the first course, of whom 12 were selected as participants, seven males and five females. There was one participant each from Kenya, Malawi, Ethiopia, Uganda, China, Viet Nam, India, Bangladesh, Myanmar and Afghanistan, and two participants from South Africa.

All participants achieved all milestones during the course up to the final module. At the final milestone, 9 months after the start of the course, 11 of the 12 participants had submitted a total of 14 research papers to international peer-reviewed journals (Table 3).^9–21 One participant completed and wrote up his project as a scientific paper, thus achieving the milestone, but was not granted permission locally to submit it to a peer-reviewed journal. Of the remaining 11 participants, three (who were OR Fellows) each submitted two papers by the deadline. Of the 14 research papers submitted, 13 had been published within 24 months following the end of the course (by 1 May 2012).^9–21 One paper failed to make progress: it was withdrawn from the first journal to which it was submitted after 9 months because of a lack of reviews. After this discouraging experience, the principal investigator did not submit anywhere else. The other two OR Fellows duly submitted their second papers before their contract end, but after the course end. These papers thus fulfilled their fellowship obligations but were not counted as course outputs.

TABLE 3.

Operational research projects completed during The Union/MSF course and publication status^*

Participant’s country	Title of research project	Name of journal	Publication status
South Africa	Recording delivery of isoniazid preventive treatment to children: operational challenges	Int J Tuberc Lung Dis	Accepted November 2009; published9
Viet Nam	Completeness and consistency in recording information in the tuberculosis case register, Cambodia, China and Viet Nam	Int J Tuberc Lung Dis	Accepted March 2010; published10
Malawi	Early active follow-up of patients on antiretroviral therapy (ART) who are lost to follow-up: the ‘Back-to-Care’ project in Lilongwe, Malawi	Trop Med Int Health	Accepted April 2010; published11
China	Performance of culture and drug susceptibility testing in pulmonary tuberculosis patients in northern China	Int J Tuberc Lung Dis	Accepted June 2010; published12
India	Tuberculosis ‘retreatment others’: profile and treatment outcomes in the state of Andhra Pradesh, India	Int J Tuberc Lung Dis	Accepted June 2010; published13
Ethiopia	Intensified tuberculosis case finding among people living with the human immunodeficiency virus in a hospital clinic in Ethiopia	Int J Tuberc Lung Dis	Accepted August 2010; published14
Kenya	Loss to follow-up from tuberculosis treatment in an urban informal settlement (Kibera), Nairobi, Kenya: what are the rates and determinants?	Trans R Soc Trop Med Hyg	Accepted August 2010; published15
India	Characteristics and programme-defined treatment outcomes among childhood tuberculosis (TB) patients under the national TB Programme in Delhi	PLoS ONE	Accepted September 2010; published16
Bangladesh	Why are tuberculosis patients not treated earlier? A study of informal health practitioners in Bangladesh	Int J Tuberc Lung Dis	Accepted October 2010; published17
South Africa	Identification of losses to follow-up in a community-based antiretroviral therapy clinic in South Africa using a computerized pharmacy tracking system	BMC Infect Diseases	Accepted October 2010; published18
Malawi	Retreatment tuberculosis cases categorised as ‘other’: are they properly managed?	PLoS ONE	Accepted October 2011; published (previously rejected by Int J Tuberc Lung Dis)19
Uganda	Loss to follow up from isoniazid preventive therapy among adults attending HIV voluntary counseling and testing sites in Uganda	Trans R Soc Trop Med Hyg	Accepted October 2011; published (previously rejected by Int J Tuberc Lung Dis)20
Viet Nam	Human resource requirements for quality-assured electronic data capture of the TB case register	BMC Res Notes	Accepted January 2012; published (previously rejected by Int J Tuberc Lung Dis)21
Afghanistan	Does task shifting in TB microscopy services to non-certified microscopists affect quality? A cross-sectional study in Afghanistan	Health Policy Plann	Withdrawn from the journal after 9 months—no further progress
Myanmar	Assessment of TB treatment outcomes in HIV-positive TB patients in an integrated HIV care programme in Mandalay, Myanmar	Not submitted to a peer-review journal

^*All research papers were submitted to a journal before April 2010, with the exception of one participant from Myanmar who was unable to submit.

The Union = International Union Against Tuberculosis and Lung Disease; MSF = Médecins Sans Frontières; HIV = human immunodeficiency virus.

Post-course research activities and qualitative outcomes

The questionnaire had a 100% response rate (12/12 participants). During the year following the end of their course, participants had undertaken and completed (or undertaken but not yet completed) 34 additional OR projects (not including course papers). Once the data about publications reported in the questionnaires had been verified by other means, it was found that course participants had written a further 19 papers during that subsequent year, mostly as a first author and some as a co-author (Table 4): 17 of the 19 papers were published within 24 months of course end (by 1 May 2012), and two remained ‘submitted’.^22–38 Participants reported 10 posters and/or oral presentations at conferences. Nine participants had been involved in OR training and/or mentoring (five as mentors in subsequent OR training activities undertaken by The Union and/or MSF), and five had reviewed papers for peer-reviewed journals.

TABLE 4.

Research activity during the year following the end of The Union/MSF operational research course

Research activity	Total n	OR Fellows (n = 5)	Non-OR Fellows (n = 7)
Project undertaken and completed, or undertaken but not yet completed	34	22	12
Paper published	17	16	1
Paper submitted, but not yet published*	2	2	0
Research presented as poster or conference presentation	10	5	5
Participant involved in training or mentoring	9	4	5
Participant involved in reviewing papers	5	5	0

^*The status of papers reported as ‘submitted’ in August 2011 in the questionnaire has been updated as of 1 May 2012 (24 months after course end).

The Union = International Union Against Tuberculosis and Lung Disease; MSF = Médecins Sans Frontières; OR = operational research.

Table 4 shows that it was almost exclusively OR Fellows who had submitted and/or published papers. Participant involvement in presenting research at conferences and being involved in training or mentoring did not differ between the two groups. However, only OR Fellows had reviewed scientific papers for journals.

Four participants reported three changes in policy and five changes in practice that had been influenced by projects they had undertaken and completed (or undertaken but not yet completed) in the year after the course. Details of the reported changes are not provided here, given the small number of identifiable participants involved in the evaluation. It would also be desirable first to validate, then to analyse and report changes using a mixed methods (qualitative-quantitative) approach.

Participants also mentioned a variety of changes in their organisation’s approach to OR (Table 5). These included changes in understanding, commitment, policy, plans, human resources trained and allocated for OR, improved data management, new partnerships and inclusion of OR in funding applications. Time and funding for OR activities were reported to be the biggest challenges for participants during the year after the course.

TABLE 5.

Changes observed by participants^* in their organisations’ approach to OR during the year following the end of The Union/MSF operational research course

Awareness, commitment, integration into guidelines and routine practice:

After the course, I published a paper. This helped my boss to understand what OR is OR has been identified as an important tool for informing policy and practice OR has now been included in our organisation’s policy, and we have an OR agenda for the coming year Research guidelines have been developed Monitoring and evaluation guidelines are being developed for the programme All staff are being encouraged to conduct and publish OR More human resources are being allocated for OR in our projects

Capacity building and communication between researchers:

Monitoring and evaluation activities are now performed more efficiently by our team. We are using EpiData† as well as spreadsheets The data management skills learned during the course enabled me to perform sophisticated data extraction and analysis; the results were presented as posters in international conferences and important programme meetings OR training for staff at provincial level has meant more human resources for OR More OR research is being conducted by junior researchers

Partnerships and communication between institutions:

Partnerships were built with the National Tuberculosis Programme and the World Health Organization for conducting OR New partnerships have been built with different institutions We are currently discussing with partners about starting to provide OR training New research partnerships have been established with several international institutions

Funding and facilitating the research process:

The budget for OR has been incorporated into all new projects and grants Resource mobilisation for OR was enhanced through our Global Fund to Fight AIDS, TB and Malaria grant I was involved in writing a proposal for funding for OR projects and training

^*Participants’ observations are not presented as direct quotes. Observations have been abbreviated and/or paraphrased, and grouped into categories to ensure anonymity.

^†The EpiData Association, Odense, Denmark.

The Union = International Union Against Tuberculosis and Lung Disease; MSF = Médecins Sans Frontières; OR = operational research; TB = tuberculosis; AIDS = acquired immune-deficiency syndrome.

Participants’ recommendations were grouped into four main categories (Table 6). They wanted to see increased awareness about what OR entails and contributes, increased commitment to the subject and more integration of OR into routine activities. For example, it should be included in guidelines, strategies, agendas, workplans, funding proposals and in monitoring and evaluation. Second, they recommended continuing to develop capacity and confidence among researchers, supported by ongoing communication. This included expanding results-oriented courses with follow-up, such as The Union/MSF model, at national, regional and local levels. They saw the need for a platform for researchers to share knowledge and establish priorities for OR. Third, participants suggested an emphasis on new and/or stronger partnerships and communication between institutions such as ministries of health, research bodies, local and international organisations. Fourth, they noted a need for funding and other types of practical support for the research process, such as facilitating ethics approval, addressing institutional barriers, and creating more opportunities to acquire and disseminate OR knowledge.

TABLE 6.

Participants’ recommendations^* for improving and expanding OR

Awareness, commitment, integration into guidelines and routine practice:

Promote and facilitate more OR Improve education for programme managers and policy makers about OR Convince managers and policy makers that OR is essential for health programmes, in particular for evaluating and revising guidelines and strategies Get OR onto national agendas Develop OR guidelines at country level and for institutions Make partnership a key element in national OR plans Include OR in the monitoring and evaluation framework of all public health programmes and projects Hold regular OR workshops to share knowledge among different people, i.e., decision makers, health workers Encourage the medical profession to have a more ‘public health’ perspective

Capacity building and communication between researchers:

Provide training courses such as The Union/MSF 1-year, 3-module, result-oriented course Provide training with follow-up and build capacity of programme staff and health care workers at national, regional and local levels Set up an internal platform for OR researchers—to answer researchers’ questions and share OR research experiences with each other Help young researchers gain more confidence in their research capabilities Share knowledge between OR course alumni; e.g., published articles Develop packages of OR that need to be conducted and provide funding

Partnerships and communication between institutions:

Build partnerships between country institutions and international institutions Develop strong OR partnerships between research institutions and the Ministry of Health at national level Develop stronger links between country institutions involved in health care Build partnerships between national universities/tertiary institutions and organisations such as The Union and MSF that are involved in OR to ensure practical knowledge is passed on from experienced colleagues to up-and-coming researchers

Funding and facilitating the research process:

Advocate for having a dedicated budget for OR in all public health programmes and projects Obtain more funding and commitment from institutional leaders Find funding for young researchers to afford online journal publication fees Advocate for efficient national research committees that are responsible for closely examining and quickly processing research submissions Facilitate ethical approval of OR projects Create more opportunities internationally and in countries for people to acquire and disseminate OR knowledge Address barriers such as difficulties gaining permission, resistance, weak partnerships, hierarchical dominance of research activity

^*Participants’ recommendations are not presented as direct quotes. Recommendations have been abbreviated and/or paraphrased in places, and grouped together and into categories to ensure anonymity.

OR = operational research; The Union = International Union Against Tuberculosis and Lung Disease; MSF = Médecins Sans Frontières.

DISCUSSION

Although it reports on a small number of participants, this evaluation shows that the Union/MSF OR course model was successful in achieving its goals. All participants reached their immediate goal of preparing a paper for submission for publication. Only one participant failed to submit a paper, and this was related to lack of local approval for publication rather than any deficiency in effort or scientific merit. The one paper that was submitted but failed to get published was related to journal choice. The journal’s unacceptable delay in reviewing demotivated the principal investigator. Course organisers and facilitators will need to work out how to minimise the risk of such situations recurring in subsequent courses. The reasons for successful submission outputs are likely to be a combination of strict criteria and careful selection of motivated candidates, strong mentorship from experienced faculty that was available during, between and after the three modules, a ‘learning by doing’ approach that accompanies participants in real time through the research process, the mutual support and healthy competition generated by having participants work to the same timeline, strict adherence to milestones that encouraged participants to stay focused on their projects, and a performance-based OR fellowship programme.

Although it cannot be concluded that any post-course activities or changes are due wholly or partly to the course, it is likely that the course played some role. Moreover, by tracking what happens after the course and learning more about what facilitates the expansion of OR and changes in policy, practice and organisational approaches to OR, we should be able to further improve OR training and strategy.

Participants trained and inspired by the course continued to pursue research activities once the course had finished. Participants published more papers, prepared posters and presentations for conferences, and participated in other activities that assist with expanding OR, such as training, mentoring and paper reviewing. It is encouraging to see that almost the same proportion of non-Fellows as Fellows were involved in presenting posters or conference presentations, training and mentoring during the year after the course. However, the large difference in publication outputs between Fellows and non-Fellows would seem to indicate that financial support, mentorship and an acknowledged responsibility to undertake and publish research contributed to the higher number of publications being produced.

A number of activities should be considered for improving and expanding OR implementation. These include 1) improving understanding of OR and including it in guidelines, frameworks, plans and routine activities; 2) expanding training that is results-oriented and that accompanies researchers right through the process of conducting OR; 3) establishing better means for communication and collaboration between researchers, partners and institutions; and 4) improved funding and facilitation of the research process.

Based on the findings of this evaluation, we will continue to routinely track 1) submissions and publications of participants’ course projects and 2) submissions, publications and related OR activities of participants following their completion of the course, including involvement in further research projects, training, mentoring and reviewing. We will also continue to collect participants’ opinions about whether any changes in policy and practice have occurred as a result of the publication of their papers. Significant examples could potentially be followed up with an in-depth analysis to document the change and to understand what facilitated getting the research into policy and/or practice.

A research alumni network is being established to collect the post-course outputs and observations. The network aims to facilitate ongoing communication between current students, graduates and more experienced researchers. By providing support, resources and incentives to produce output during and after the course, the Union/MSF initiative aims to contribute to an enabling, informative and motivating environment for OR—long after participants achieve their ‘paper submitted’ milestone.

This evaluation has some clear limitations. Notably, it involved a very small number of participants: 12 people from the first course. Nevertheless, as an initial analysis of a new approach to teaching OR, it provides support for continuing the model. Since the first course in June 2009, five more courses have been completed and six are currently in progress. Subsequent courses, although not yet analysed in the same way, appear to be generating similar outputs and will be reported on in due course.

In conclusion, the Union/MSF OR course and fellowship programme was successful not only in having participants complete the final milestone of submitting a paper for publication, but also in training and helping to motivate them to continue working in OR after the course.