How do we measure the success of operational research?

A number of training initiatives are aimed at expanding operational research (OR) in health. The Operational Research Assistance Project (ORAP) implemented by the Desmond Tutu TB Centre (DTTC) aimed to strengthen the capacity of South African professionals to conduct operational research (OR) as an integral component of health programmes in order to contribute to the improved quality and performance of the health system. The project was undertaken in partnership with the TREAT TB team of the International Union Against Tuberculosis and Lung Disease (The Union) and in collaboration with the South African National Department of Health (DoH) and the US-AID Mission to South Africa.

Research teams established in each of the nine provinces in South Africa consisted of a health service provider from the DoH and an academic researcher from a local academic/research organisation. In addition, young researchers from the Department of Paediatrics and Child Health, Stellenbosch University, could also enrol. Research topics were selected from the National Health Research Priorities list that is developed by the National DoH in collaboration with key stakeholders. Over an 18-month period, teams were guided through the process of developing ethically approved OR protocols, undertaking the research (with funding provided to support implementation), managing and analysing the data, disseminating the results and writing a manuscript for publication in a peer-reviewed journal. A senior DTTC staff member was assigned to mentor each team throughout the process.

There were many measurable successes: 103 health service providers and medical researchers from all nine provinces of South Africa were trained, and a manual ‘Operational research to improve health services: a guide for proposal development’¹ was developed, distributed at The Union World Conferences on Lung Health and translated into Spanish for broader distribution by TREAT TB.

To date, 36 research studies have been undertaken and 20 posters and 14 oral presentations have been presented at conferences. Thirteen manuscripts have been submitted and/or published, 16 are being prepared and eight projects are still underway.

Interviews conducted during an external qualitative evaluation with a range of project stakeholders, including project partners, mentors, project participants and DoH managers, highlight the range of benefits that participants gained from ORAP. Key benefits included improved knowledge and understanding of OR, skills gained in designing and implementing OR, improved time and project management, budgeting and problem-solving skills.

Participants from academic institutions said that ORAP provided them with an opportunity to gain understanding of the DoH tuberculosis programme and that publishing opportunities helped to develop their careers. Participants from the DoH reported that they had greater insight into their work when evaluating tuberculosis data and that they could better identify problems existing in the health service. They also felt that ORAP built an improved relationship between academic institutions and the DoH.

ORAP has presented several challenges. The mentorship needs of the participants were far greater than anticipated, and the limited pool of experienced mentors could not always meet the needs of the research teams. The differing skill levels within the research teams, time constraints, different research interests and lack of physical proximity between team members created tensions that sometimes prevented effective collaboration.

Although several published manuscripts have made recommendations for health service improvements, none have been implemented to date. The involvement of service providers has not yet facilitated the process of research translation. The question of whose responsibility it is to ensure that action is taken in response to the findings needs to be addressed and the role of academic researchers in this process clarified.

The allocation of financial resources is an important measure of the priority placed on OR. Despite positive feedback from the DoH, no resources have been allocated to fund future similar OR activities. Although the cost of OR is lower than for other research activities, OR must be appropriately costed and resources allocated to ensure sustainability.

We recognise that OR initiatives need to be measured against multiple outcomes. While the number of published manuscripts is an important measure, this does not suffice on its own. The contribution and value of capacity building and skills improvement have the potential for a broader effect beyond OR, but these, too, are difficult to measure. Ultimately, the impact on policy and or practice is important, and when and how to measure this remains a challenge.