What happens after participants complete a Union-MSF structured operational research training course?

Abstract

Setting: Eight operational research (OR) courses run by the International Union Against Tuberculosis and Lung Disease (The Union) and Médecins Sans Frontières (MSF) for participants from low- and middle-income countries. There is a knowledge gap about whether participants continue OR after course completion.

Objectives: To determine 1) the research output of participants and their institutions after course completion; 2) the influence of OR fellowships on output; and 3) the output of non-OR fellows stratified by sex, region and staff position.

Design: A self-administered e-mail questionnaire survey.

Results: Of 83 participants who completed a course, 76 (92%) responded to the questionnaire. Following course completion, 47 (62%) participants completed new research projects, 38 (50%) published papers (vs. 25 [33%] who had published before the course), 42 (55%) presented posters or oral abstracts at conferences, 33 (43%) facilitated at further OR courses, 29 (38%) reviewed scientific papers, 25 (33%) secured further OR funding and 55 (72%) said their institutions were involved in OR implementation or capacity building. OR fellows performed better than non-OR fellows. Among the latter, males and participants from Asia had better output than females and participants from Africa (P < 0.05).

Conclusion: The significant proportion of participants continuing to engage in OR after course completion provides encouraging evidence of the long-term value of this capacity building model.

The International Union Against Tuberculosis and Lung Disease (The Union) and Médecins Sans Frontières (MSF) started a programme of capacity building in operational research (OR) in 2009, and in 2012 joined forces with the Special Programme for Research and Training in Tropical Diseases (TDR), hosted at the World Health Organization. The aim of this joint programme, known as the Structured Operational Research and Training Initiative (SORT IT), is to improve health care delivery and public health through OR. This is done through integrated OR training courses in which participants from low- and middle-income countries are enrolled and taken through a research project from protocol development to implementation, data collection, analysis, and the writing and submission of a paper to a peer-reviewed journal.¹

We carefully monitor output from each course, including 1) the number and percentage of enrolled participants who complete the course, 2) the number of scientific papers submitted and published in peer-reviewed journals, and 3) whether there has been any impact of the research study on policy and practice. We are also interested in knowing whether there has been any continuation of research activities after the course has been completed and whether the skills learnt during the course have been used to expand the discipline of OR. An assessment conducted 1 year after the end of the first OR course found that the 12 course participants had continued with research activities, submitting and/or publishing a total of 19 papers, presenting posters at conferences and, in some cases, participating in training courses, mentoring and reviewing scientific papers in peer-reviewed journals.²

Apart from the study cited above, no other publications have reported on research activities of OR course participants after completion of training. We therefore decided to follow up participants who had completed our first eight OR training courses, and report on research-related output since the courses were completed. Specific objectives were to determine 1) the research output of participants and their institutions after course completion, including a comparison with papers published by participants before the course; 2) the influence of OR fellowships on their output; and 3) the output of non-OR fellows stratified by sex, region and staff position.

METHODS

Study design

This was a cross-sectional study using self-administered questionnaires.

Setting

The structure and organisation of the OR courses have been described elsewhere.^1,2 In brief, each course is run over 8–12 months, with Module 1 (protocol development) leading to Module 2 (data collection and analysis) and Module 3 (paper writing). Because of the tight timeframe and ethics considerations, participants are usually selected only if routine data from government or non-governmental health facilities are already available for collection, cleaning and analysis. Four important milestones have to be achieved for participants to remain on the course. Failure to meet the first three milestones results in termination of attendance at the course, while non-submission of the final paper to a peer-reviewed journal within 4 weeks of completion of Module 3 results in the participant being declared a ‘course failure’.

Some outstanding participants who complete the course may be offered full- or part-time OR fellowships with both The Union and MSF. Fellows work in the field in close collaboration with government programmes or non-governmental organisations and are mandated to lead on OR activities in the countries in which they are based.^3,4 They are placed on 12-month contracts and are required to submit a minimum of two papers to peer-reviewed journals each year to continue in their positions. Both fellows and non-fellows are encouraged to mentor and facilitate future courses and to assist in peer review of OR papers submitted by other authors to national or international journals.

Participants

Study participants included all those who had successfully completed one of the eight courses. Details regarding the courses and course participants are shown in Table 1.

TABLE 1.

Details of the eight OR courses and participants

Source of data, variables and data collection

A formal questionnaire was developed, pilot-tested among a few participants and OR fellows, and then finalised (Table 2). It was sent by e-mail on 12 and 13 June 2013 to study participants who had completed the courses. A reminder was sent 1 month later to 63 participants who had failed to respond. From 16 September to 1 November 2013, further reminders were sent to 22 participants who had failed to respond. The survey was closed at the end of November 2013.

TABLE 2.

Survey questionnaire sent to study participants who had completed the courses

All questions were related to output after completion of the course. All required simple yes/no answers, and more detailed information was requested when the answer was affirmative. Questions related to the completion and publication of any research studies (the latter constituting any published paper the participant had co-authored), poster or oral abstract presentations at national or international conferences, mentoring or facilitating at OR courses, conducting peer-review of scientific papers for journals and obtaining any funding for OR. One final question was related to whether the institution where the participant worked at the time of doing the course had initiated any OR implementation or capacity building.

Further information about participant characteristics was obtained from the course files. For each participant, any research paper published before the course began was obtained through a Medline search; all papers that participants stated in the questionnaire as having published after the course were also verified on Medline.

Analysis and statistics

Data from the completed questionnaires were entered into an MS Excel file (Microsoft, Redmond, WA, USA). An analysis was carried out using MS Excel and Open Epi Software (Emory University, Druid Hills, GA, USA). Research output was analysed according to whether participants were OR fellows or non-OR fellows, and among the latter by sex, region and staff position at the time of participating in the course. Differences between groups were compared using the χ² test for categorical variables and the Wilcoxon rank-sum test for continuous variables, with levels of significance set at 5%.

Ethics

The Ethics Advisory Group of The Union, Paris, France, determined that ethics clearance was not required for this study.

RESULTS

Participants and characteristics

Of 93 participants enrolled in the eight courses (Table 1), 83 (89%) completed the course, met all four milestones and submitted a paper to a peer-reviewed journal. Ten participants failed to complete the course: four were unable to obtain local ethics approval for their studies, four failed to collect data and achieve milestone 3, and two failed to achieve the final milestone and submit their papers on time. Of those who completed their course, 76 questionnaires were returned (response rate 92%).

Of the 76 participants who returned the questionnaires, 33 (43%) worked in the government health sector, 28 (37%) worked in non-governmental organisations (NGOs) and 15 (20%) were university- or college-based. At the time of starting the course, 42 (55%) were medical doctors; 11 (14%) were paramedical clinical officers; 9 (12%) were monitoring and evaluation officers, usually with the government sector; 6 (8%) were research officers; and 4 (5%) were nurses. The remaining four participants were a pharmacist, a nutritionist, a laboratory technician and a communication/advocacy officer.

Research output

About 60% of the participants had completed further research projects and 50% had published papers other than for the course project (Table 3). Over 50% had presented at conferences, just over 40% had mentored or facilitated at OR courses and nearly 40% had acted as peer-reviewers for scientific papers. One third of participants had obtained new funding for OR, and >70% of the participants stated that their institutions were involved in OR implementation or capacity building. The median time of follow-up (i.e., from course completion to completion of the study questionnaire) was longer for OR fellows than non-OR fellows. With every individual research output, OR fellows performed better than non-OR fellows (P < 0.05), although there was no significant difference with respect to institutional OR capacity building.

TABLE 3.

Individual and institutional research output following successful completion of an OR course, stratified by whether or not the participant was an OR fellow at the time of the study

Publications

Of the 76 participants, 25 (33%) had had an article published before attending the course, significantly fewer than after the course (n = 38, 50%, P = 0.03). Altogether, 75 papers had been published by participants before the course. Of the 25 (32%) participants with previous publications, 10 were first authors. Of the 10 OR fellows, 5 (50%) had published before the course, with one listed as first author. Of the 66 non-OR fellows, 20 (30%) had published before the course, with 9 listed as first authors.

All papers published by OR fellows were validated on Medline. Of 110 papers published by non-OR fellows, 90 were validated; however, 20 (18%) could not be validated for the following reasons: the journal was not identified on Medline (n = 9), the article was not found on Medline (n = 4) or too few details were provided to be able to identify the article (n = 7).

Research output of non-OR fellows

Altogether, non-OR fellows published 110 papers; the journals in which their work was published are listed in Table 4. Research output among non-OR fellows, stratified by sex, region and whether or not participants were in a research-related post at the time of the course is shown in Tables 5, 6 and 7, respectively. The only significant differences were that a higher number of research projects were completed and papers published by male compared with female participants (Table 5); and a higher number of Asian than African participants completed research projects, published papers, mentored at training courses and peer-reviewed scientific papers (Table 6). There was no difference in research output in relation to whether or not the participant occupied a research-related post at the time of the course.

TABLE 4.

Journals in which non-operational research fellows have published their research studies

TABLE 5.

Individual and institutional research output among non-OR fellows after successful completion of an OR course, stratified by sex

TABLE 6.

Individual and institutional research output among non-OR fellows after successful completion of an OR course, stratified by participants' region ^*

TABLE 7.

Individual and institutional research output among non-OR fellows after successful completion of an OR course, stratified by whether or not participants were in a research-related post at the time of the course

DISCUSSION

This study found a considerable amount of research activity from a large number of participants who had completed one of the eight OR courses run by The Union and MSF. OR fellows performed particularly well, unsurprisingly, as they are mandated to undertake, promote and lead on OR, and their 12-month contracts are performance-related and renewable only if at least two papers are submitted to journals within the set time period. Significantly more participants published papers after than before the course. It was most encouraging to see substantial output from non-OR fellows, with between one third to a half completing and publishing research projects, presenting at conferences and reviewing papers for scientific journals. This group published over 100 papers in a wide variety of journals. Finally, just over a quarter of non-OR fellows had secured funding for OR, and there seemed to be good commitment from the institutions to which they belonged.

In terms of completed projects and papers from non-OR participants, males performed better than females and participants from Asia did better than those from Africa. This may be due to the active involvement of the South-East Asian Union Office, with two OR fellows implementing three OR courses annually (two in India and one in Asia) and developing collaboration with India's Revised National Tuberculosis Control Programme and medical colleges in India to expand the discipline of OR.

This is the first study to assess post-course output from a large number of participants. The strengths were the good response rate, of over 90%, from participants completing the course and the use of a broad range of research output (studies completed, papers published, poster and oral abstract presentations, peer review of scientific papers for journals and acquisition of research funding) to assess the amount of research activity following each course. We were also able to validate the lists of published papers in most cases.

The study also had several limitations. First, the questionnaire was self-administered and some of the answers lacked detail. Second, because of some ambiguity in the question related to OR uptake by the participants' institutions, we were unsure whether the answers reflected the institution's engagement in OR before or after the course, thus diminishing the value of this response. Third, the time periods were different for participants, with some completing the questionnaire 3 years and some 6–9 months after course completion. Fourth, as the survey questions were framed, they did not allow us to determine whether the research activities in question were actually initiated before or after the course. Other than published papers, we were therefore unable to collect data on other types of research output by participants before attending the course. Such data would have enabled us to compare ‘before’ and ‘after’ output and better assess the overall effect of the course. Fifth, it would have been useful to compare research output from those who completed the course with those who failed the course. We did in fact attempt to do this, but only received two responses from the 10 who failed; this aspect of the study was therefore abandoned. Sixth, the questionnaire was not designed to capture reasons why some participants chose to engage in further research while others did not, and this is an area that merits further study. Finally, it would also have been valuable to monitor the impact of post-course research on policy and practice; however, this was beyond the scope of the current study.

Despite these limitations, the survey provides a useful overview on post-course research activity. In the courses, we aim to teach participants practical skills required for conducting and publishing OR. While the focus is on one research project to be completed and written up within the time frame of the course, our findings suggest that about half of the participants engage in further OR after the course has finished, possibly as a result of inspiration, enthusiasm, interest and knowledge gained during the course. This is particularly encouraging, as the majority of our participants were health programme implementers working in government or NGO sectors, many of whom had never conducted research in the past.

The survey has also given us better insight into how to collect data in the future. We have revised the questionnaire to 1) remove ambiguity, 2) instil better clarity, 3) enquire about research activities and output before the course, 4) ensure that enquiries into output after the course are related to those research activities that were initiated after the course, and 5) obtain more details about post-course research activities. We now propose to send the questionnaire to course participants between 15 and 18 months after they have completed their courses so that the follow-up period is the same for all participants. When participants enrol in these courses, we will engage them in an alumni network from the start, which might help in reducing the workload during the follow-up stage.

In conclusion, we have found that a significant proportion of participants successfully completing our OR courses continue to engage in OR. This is encouraging, especially as the majority have been health programme implementers. This adds long-term value to the actual course. We aim to continue assessments of course participants, but in a more formal and structured way.