The Structured Operational Research and Training Initiative for public health programmes

Abstract

In 2009, the International Union Against Tuberculosis and Lung Disease (The Union) and Médecins sans Frontières Brussels-Luxembourg (MSF) began developing an outcome-oriented model for operational research training. In January 2013, The Union and MSF joined with the Special Programme for Research and Training in Tropical Diseases (TDR) at the World Health Organization (WHO) to form an initiative called the Structured Operational Research and Training Initiative (SORT IT). This integrates the training of public health programme staff with the conduct of operational research prioritised by their programme. SORT IT programmes consist of three one-week workshops over 9 months, with clearly-defined milestones and expected output. This paper describes the vision, objectives and structure of SORT IT programmes, including selection criteria for applicants, the research projects that can be undertaken within the time frame, the programme structure and milestones, mentorship, the monitoring and evaluation of the programmes and what happens beyond the programme in terms of further research, publications and the setting up of additional training programmes. There is a growing national and international need for operational research and related capacity building in public health. SORT IT aims to meet this need by advocating for the output-based model of operational research training for public health programme staff described here. It also aims to secure sustainable funding to expand training at a global and national level. Finally, it could act as an observatory to monitor and evaluate operational research in public health. Criteria for prospective partners wishing to join SORT IT have been drawn up.

Operational research (OR) can be defined in various ways, but from a public health perspective it has been defined as research into strategies, interventions, tools or knowledge that can enhance the quality, coverage, effectiveness or performance of the health system or disease programme in which the research is being conducted.¹ We view operational research as a spectrum of activities that encompasses reviews of data already collected in patient registers, treatment cards or patient files, evaluations of operational practices or the implementation of new strategies and technologies.

Operational research is often observational in nature, and can involve descriptive or cross-sectional studies, retrospective or prospective cohort studies and sometimes case-control studies. Recent guidelines for the reporting of observational studies (the STROBE statement) provide a logical structured roadmap for this type of research, thereby improving its scientific credibility.² Operational research also needs to be conducted within a sound ethics framework that includes the principles of informed consent and data confidentiality, and in all cases, study results should be fed back to the local communities in an accessible and understandable manner.³

The key role of operational research in improving health programme performance is well-recognised,¹ and the subject is consequently strongly promoted by donors and technical agencies. For example, The Global Fund to Fight AIDS, Tuberculosis and Malaria (the Global Fund) recommends an allocation of up to 10% of the total grants towards monitoring and evaluation including operational research. Despite this potential opportunity for programme strengthening, the implementation of operational research is weak in many of the low- and middle-income countries that are in most need of it, and lack of operational research capacity is one of the main reasons behind this.¹

In March 2009, the International Union Against Tuberculosis and Lung Disease (The Union) and Médecins sans Frontières Brussels-Luxembourg (MSF) began to develop a programme for integrated operational research and training that not only teaches the principles of the what, why and how of operational research, but also incorporates the development, implementation and writing up of a research project as an integral part of the programme. Training is combined with practical aspects, and the success of a participant is judged on whether or not a research project has been designed and completed, with a paper submitted to a peer-reviewed journal within the stated time frame of the programme.^4,5 The organisations also track whether these papers have been published and whether the research has had any impact on policy and practice.

In January 2013, The Union and MSF joined forces with the Special Programme for Research and Training in Tropical Diseases (TDR), which is based at the WHO, and the three organisations have developed a blueprint for training public health programme staff under the Structured Operational Research Training

Initiative (SORT IT). The overarching goal of SORT IT is to improve health care delivery and public health through operational research. The main objectives are to help countries to:

• 1)conduct operational research in accordance with their own priorities
• 2)develop adequate and sustainable operational research capacity in public health programmes, and
• 3)create an organisational culture of policy and practice that is informed by operational research and leads to improved programme performance.

The structure and key components of a SORT IT programme, based on the format that The Union and MSF have been developing since 2009, are presented below.

THE SORT IT PROGRAMME OF INTEGRATED OPERATIONAL RESEARCH AND TRAINING

Selection of participants

Several months before the programme starts, a programme brief and application form are disseminated to the collective networks of the three organisations, including alumni networks from previous research programmes. Applications are received by a specified date, and strict selection criteria are imposed (see Table 1). Such criteria are essential to ensure full commitment by the applicant and his/her institution to the expected goals of the training, and to optimise the chance of successful completion of the course. The 12 participants selected for the programme receive a full scholarship including tuition fees, travel expenses, full accommodation and the costs of subsequent open access publication.

TABLE 1.

Selection criteria for candidates for SORT IT integrated operational research and capacity building in public health programmes

Research projects

In their application forms, the applicants are asked to formulate and submit a research question based on a perceived problem or constraint within the programme or health system in which the applicant works. This is a critical way of ensuring programme relevance and participation with the proposed research by programme managers and supervisors. As research studies have to be carried out within a 9–10 month timeframe, including the ethics review of the protocols, simple projects are encouraged. Those based on monitoring and evaluation data already collected and available in registers, treatment cards, patient files or electronic data sets are most suitable. Prospective studies involving patient interventions and patient questionnaires are less likely to be accepted, as they cannot usually be completed within the relatively short time frame of the programme. Relying on existing data records also encourages health workers and programme staff to see the utility and importance of routine programme data. The saying ‘data rich, information poor’ is applicable to many low- and middle-income countries, and is one that needs to be replaced by a new adage of ‘data and information rich’.

Programme structure and milestones

The structural details of the programme comprising the three workshops are highlighted in Table 2. Workshop 1 concentrates on development of study protocols, with due attention to ethics; Workshop 2 focuses on electronic quality-assured data capture and analysis using open and free access software (EpiData software, EpiData Association, Odense, Denmark – freely available at http://www.epidata.dk), with the data workshop tailored to meet the data capture requirements of individual study protocols; Workshop 3 focuses on writing up the paper, providing guidance on how to submit the paper to a peer-reviewed journal and how to handle peer review, and finally on promoting and assessing the effects of the research on policy and practice. The training format includes lectures, break-out mentor groups and plenary sessions where participants present their work (protocols and papers in progress) to the whole group and receive feedback from their peers and facilitators. Milestones must be achieved to remain in the programme, and to receive a programme completion certificate all milestones must be completed, including the final submission of a paper to a peer-reviewed journal (Table 3).

TABLE 2.

Outline of SORT IT integrated operational research and capacity building in public health programmes

TABLE 3.

Milestones for SORT IT integrated operational research and capacity building in public health programmes

Mentorship

Strong, hands-on mentorship is a key feature of the course, and this is provided by the facilitators. For Workshop 1 and 3, there are usually eight facilitators, who work in pairs mentoring three participants in each group. Each pair has a senior and a junior facilitator, the latter ideally being a participant who has distinguished himself/herself in a previous programme and is now learning the art of mentoring by being paired up with an experienced senior facilitator. For Workshop 2, there are usually four to six facilitators, with one allocated to a group of two or three participants. The facilitators help the participants develop their protocols and papers through iterative, side-by-side teaching that involves the sharing of versions of the protocols and papers through e-mail exchange. Between workshops and after Workshop 3, the facilitators stay in touch with their participants to help them with data collection, data analysis, paper writing, peer review, paper revisions and, in the event of rejection, preparation of the paper for a different journal.

Monitoring and evaluation of the workshops and programmes

Every workshop is formally appraised by the 12 participants with a structured questionnaire combining closed and open questions so that participants can indicate how the programme or its components can be improved. For example, the first two workshops used to be separated by a gap of 2 months, but it became clear after the first 2 years that in most settings it was better to have them back-to-back, so that the electronic data capture instrument could be firmly linked to the protocol immediately after it was written and the costs of facilitator and participant travel could be reduced.

There is also a structured approach to monitoring and evaluating the programme outputs. The first programme was started in Paris, with Workshop 1 in August 2009 and ended with Workshop 3 in March 2010. By 31 January 2014, 17 additional programmes had been held in Paris (n = 3), Luxembourg (n = 3), Fiji for the South Pacific region (n = 3), India (n = 3), Nepal for the Asian region (n = 2), Kenya and Ethiopia for the Africa region (n = 2) and Estonia for Eastern Europe (n = 1), with a total of 212 participants enrolled from 60 countries around the world. Of the different research projects undertaken, about 70% were related to tuberculosis and HIV/AIDS. However, there is an increasing focus on other areas such as malaria, maternal and child health, nutrition, smoking, non-communicable disease and neglected tropical diseases. The SORT IT initiative will expand the range of addressed public health problems.

Results of completed training programmes

The results of 14 completed programmes are shown in Table 4. Of 165 participants starting the 14 programmes, 19 failed to complete the training programme, the reasons including: failure to obtain local ethics permission for the research (n = 3); change in jobs halfway through, with the new employer reluctant to allow the participant to continue (n = 2); failure to meet the third milestone, one due to severe illness and one due to death (n = 11); failure to work with co-authors to write up the paper in time (n = 2); and refusal from the participant's institution to allow the written paper to be submitted to a journal (n = 1). We expect that over 80% of the papers submitted to peer-reviewed journals will be published within 18 months of first submission, and we target open-access journals. Publication is an important measure of the quality of research and a good indicator of a successfully completed research study. It also allows dissemination of the research globally, can influence guideline development and finally brings credibility to the authors and their institutions.⁶ The costs of open access publication are included, whenever possible, in the programme budget.

TABLE 4.

Output from nine completed operational research courses run by the Union and Médecins Sans Frontières (from April 2009 to January 2014)

In addition to publications, we also monitor and track whether the research can influence policy and practice according to criteria we have already developed.⁷ The research based on the use of programme data is already having benefits for policy, practice and advocacy in diverse areas such as attributable deaths from smoking in India,⁸ improving the care of victims of sexual violence⁹ and reducing maternal mortality.¹⁰

Improvements to the programme over time

As a result of experience and feedback from participants, several innovations have been introduced to the training programme since 2009.¹¹ New material has been developed to train participants in how to conduct literature searches and use open access reference manager tools. In addition to hard-back course folders and CD-ROMs, in which all the materials of the programme are provided, the materials are also shared via Dropbox, a web-based file sharing service, and participants are taught how to use this. We are experimenting with SurveyMonkey to perform our end of workshop evaluations. Instead of using general application forms for ethics review by the Union's Ethics Advisory Group, we have redesigned forms especially for record reviews, and these are much easier and more straightforward to complete. In the decentralised courses run in Africa, Asia and the South Pacific, two major challenges have been irregular power supply and poor/intermittent internet access. To resolve these issues, we have developed a dedicated ‘Network Kit’ and an alternative power source,¹² and these two innovations allowed our most recent Africa workshop in Ethiopia to run smoothly despite frequent power interruptions and internet failures during the week.

Finally, we are connecting our research alumni in a web-based and e-mail network so that we can track better what happens after completion of the course in terms of continued research outputs, mentorship activities and organisational changes within the participants' institutions.⁵ The creation of a research alumni community fosters global networking and could eventually evolve into a web-based observatory of operational research for low- and middle-income countries. Alumni will also be automatically linked to the larger and more diverse TDR alumni network. Both The Union and MSF offer a fellowship programme to promising participants who emerge from these programmes. This fosters long-term retention and career opportunities, including pursuit of a PhD in operational research.^4,5 SORT IT intends to develop this further starting in 2014.

CHALLENGES AND WAY FORWARD FOR OPERATIONAL RESEARCH TRAINING

Despite the impressive outputs and achievements, there have also been a number of challenges. These include: 1) coping with the high applicant demand; 2) sustaining the necessary funding; 3) difficulties encountered with ethics review boards understanding the specificities of operational research; 4) finding mentors who have both research skills and operational experience and who are prepared to work in a hands-on manner with participants; 5) pressure from participants to extend the timeframe for milestones; and 6) bearing the upfront costs for open access publications.

The demand for operational research and related training for public health programme staff far outstrips supply: for each of the last four programmes there have been 100 or more applicants for 12 places. Furthermore, in the last 2 months, requests have been received from Latin America and the Middle East for programmes to be started there. Countries are requesting this assistance, and there is a need to build sustainable, high quality national capacity in operational research in all countries to support public health programmes. It is also recognised that the basic operational research training for public health programme staff is one part of a broader need. This broader need would include more advanced methods of operational research such as modelling, economic analyses and pragmatic programme based experimental designs.

To overcome some of these challenges, SORT IT will 1) advocate for this integrated programme that has been developed over the last 4 years, 2) obtain funding from multilateral and bilateral donor organisations to implement programmes, and 3) work with other organisations and institutions to extend the programmes to other regions and countries in the world. SORT IT could also act as an international observatory to monitor and evaluate operational research done in public health.

We therefore welcome other organisations and institutions interested and experienced in operational research and capacity building to share their experiences with us and to consider joining this initiative. However, to ensure the continued quality and product-orientation of the platform for those interested in joining, SORT IT activities will need to conform to a blueprint (Table 5). Furthermore, we expect targets to be reached in terms of 1) participant evaluation of the courses, 2) the proportion of participants achieving the final milestone, 3) the proportion of papers that are published within 18 months of submission and 4) an assessment of the effect of the research on policy and practice—the 80-80-80-80 target (see Table 6). Achievements against these targets will allow different members of the SORT IT consortium to audit their capacity building initiatives, and if achievements fall below the thresholds a quality improvement plan will need to be implemented so that standards can be raised. Membership of SORT IT will be dependent upon the commitment of organisations to this blueprint and to integral monitoring and evaluation of output against the targets.

TABLE 5.

Membership conditions for organisations providing SORT IT-branded integrated operational research and capacity building in public health programmes

TABLE 6.

Targets to be met for organisations to remain within the SORT IT-branded integrated operational research and capacity building programme for public health staff

CONCLUSION

This year the World Health Report 2013 is entitled ‘Research for Universal Health Coverage’. The report will emphasise that good quality health care cannot be achieved without research and that all countries need to be producers and consumers of research, with research capacity strengthened not only in academic centres but also in public health programmes close to the supply and demand side of health services. This is where the integrated operational research and capacity building model that we have described fits in. The model works: it delivers outputs in terms of scientific papers, and from the evidence we have so far, it can make a difference to policy and practice. The creation of SORT IT, a coalition under the leadership of TDR, has the potential, if adequately supported by donors and technical agencies, to expand the capacity of low- and middle-income countries to undertake operational research, for the improvement of public health programmes and ultimately the health of populations. We invite interested organisations to contact TDR.