Applying DOTS principles for operational research capacity building

The principle of DOTS, a standardised system that in the field of tuberculosis relates to case finding, providing treatment, defining treatment outcomes and rigorously monitoring the effects of the interventions in cohorts of patients, has served the management of tuberculosis programmes well, and it is being applied successfully to the management of HIV/AIDS (human immunodeficiency virus/acquired immune-deficiency syndrome)¹ and non-communicable diseases.² We are now also applying the DOTS principles to operational research capacity building courses, where participants learn about operational research while at the same time doing an operational research project.^3,4

Participant selection (case finding)

We have a standardised method for selecting participants for the operational research courses we run, with strict criteria that need to be met before candidates can be considered for the scholarships covering their attendance at the three modules that comprise a course.

Operational research training (treatment)

Just as anti-tuberculosis treatment is given over a period of 6 months and has two phases—a 2-month initial phase followed by a 4-month continuation phase —the operational research course is given for a period of 8–12 months, with Module 1 (protocol development) leading to Module 2 (data collection and analysis) leading to Module 3 (paper writing). These are all 5-day modules that are taught in a standardised way (formal lectures, one-to-one mentorship for participants and group plenary sessions for participants to present their evolving work). Participants cannot proceed to the next module without having successfully completed the previous one.

Milestones and course completion (treatment outcomes)

In anti-tuberculosis treatment, sputum smear examinations at the end of 2, 5 and 6 months provide a yardstick to identify progress and pull out patients who are likely to fail or have failed first-line treatment so that they can be investigated for drug-resistant disease and changed to a more appropriate anti-tuberculosis treatment regimen. Of the cohort of persons who start treatment, the term ‘cure’ is applied to those who complete treatment in whom sputum smears are negative at the end of treatment. Death from any cause, loss to follow-up and failure are all adverse outcomes that essentially characterise those who have not or are not known to have finished their course of treatment.

The operational research course is similarly monitored by determining whether participants reach the milestones at the end of and in between modules. For example, non-submission of research protocols and completed ethics forms to the Union Ethics Advisory Committee within 2 weeks of Module 1, and inability to provide hard evidence of completed data collection between Module 2 and 3, essentially mean failure during the course and discontinuation of the person’s participation in the course. While this may appear hard, it helps to identify participants who may not be motivated to continue with operational research, and thus saves unnecessary expenses related to bringing such persons back to the next module. In the same vein as ‘cure’, successful outcome of the course is submission of an operational research paper to a peer-reviewed journal within 4 weeks of completion of the final module. As negative sputum smear examinations are an essential part of the definition of ‘cure’, our validation of this milestone is an electronic confirmation of receipt of the submitted paper by the journal. Only those participants who attain this final milestone are awarded a course certificate.

Monitoring, recording and reporting

The undeniable strength of DOTS is the rigorous monitoring of treatment outcomes of all tuberculosis patients registered in a 3-month cohort for treatment: the so-called ‘quarterly analysis of treatment outcomes’. This enables the programme and any outsider to assess the effectiveness of treatment, identify deficiencies and hopefully find solutions. The same principles are applied to the Operational Research Capacity Training Courses. All participants who start the course are assessed according to whether or not they have completed the final milestone: hence two outcomes, success or failure.

However, this is not the end. Of all papers submitted, we also assess how many are published, and regard this as an indicator of the quality of the research. This assessment is done in two ways: first, there is a cumulative assessment every 3 months in line with our quarterly reports to donors. For example, by 30 September 2012, we had completed six courses, from which 68 papers had been submitted to scientific journals; of these, 39 (57%) had been published or were in press. However, three of the courses, in Paris, Luxembourg and Fiji, only finished between July and September, and expecting papers to be accepted or published within this short time frame is unrealistic. Second, we also adopt the cohort 12-month and 24-month outcome assessment, whereby papers submitted by a certain date are assessed for their publication status 12 and 24 months after submission. We can currently do a 12-month outcome analysis on three of our operational research cohorts, with a total of 35 submitted papers, for which publication success is 27 (77%). We can only do a 24-month assessment on our first operational research cohort, with a total of 14 papers, for which publication success is 13 (93%). As time goes on, these 12- and 24-month cohort analyses will have ever larger denominators.

Operational research

The enormous amount of data collected at the tuberculosis programme level enables useful and simple operational research to be conducted with the aim of improving programme performance and treatment outcomes. As I D Rusen aptly commented last year, we also need ‘operational research on operational research’.⁵ We are currently attempting to do this: we are trying out new paradigms of training; we are assessing how best to systematically measure the effects of research on influencing policy and practice and improving health outcomes;⁶ and we have developed a network of research alumni and are annually assessing what happens to the participants in the context of operational research after completion of the training courses.⁴

In conclusion, we firmly believe that this standardised approach, with rigorous monitoring and evaluation and the testing of new initiatives, is a way forward because it provides accountability and measurement of efficacy of the operational research training programmes. Furthermore, it allows donors and other supporters to see whether the outputs justify the inputs and whether there is value for money. With a bit of creativity, this approach could also work for other health care training initiatives.