The need to improve quality, rigour and dissemination of operations research

Operations research is essential for developing a strong knowledge base and identifying innovative strategies to improve performance of health programmes. We applaud John Walley et al.’s call for operations research designed from the perspective of the in-country decision-makers and agree with the approaches outlined in this editorial.¹ However, in addition to the factors discussed, we think that there should also be additional focus on the quality of operations research and the dissemination process of findings from such research. This has tremendous implications for the importance of operations research technology transfer to the national level.

Although the importance of quality in research might seem obvious, we have found that quality and methodological rigour are often lacking. To identify proven strategies aimed at improving routine immunization services in developing countries, we recently conducted a literature review assessing both results and methodological rigour. The lack of quality and rigour for most studies and the overall paucity of well-conducted published studies was striking, especially in light of the longstanding Expanded Programme on Immunization (EPI) and the widely recognized importance and cost-effectiveness of achieving high population coverage with vaccines.²

For our study, we conducted an exhaustive literature search of published and grey literature from 1975 to 2004 using multiple databases and 41 search terms (e.g. immunization, vaccination, developing countries) and identified more than 9000 papers and manuscripts relating to routine immunizations in less-developed countries.³ Of these, only 60 papers described results or lessons learned from a specific routine immunization strategy, thus meeting our inclusion criteria. The quality and rigour of these 60 papers were then evaluated by two reviewers using a standardized assessment instrument (except in the case of observational studies where only one reviewer was used). A rating was assigned based on the presence of elements considered critical to the scientific quality of the study and to the reviewers’ ability to appropriately understand and interpret the intervention and its reported impact. Papers received lower scores if the reviewers thought that there was an inadequate description of the methodology (39%); inappropriate selection of comparison groups (28%); inadequate sample size (41%); incomplete discussion such as lack of comparison with other study results (59%) or not addressing limitations (48%); inadequate analysis such as not accounting for confounding factors (54%); or the inability to attribute results to the intervention (28%). Through this study, it became evident that there is a need for improved research quality.

We agree with Walley et al. that research should ideally focus on the operational needs of the countries and that this paradigm is sometimes challenged by realities of limited public health resources and competing interests. Operations research itself is often not a main concern nor an area of expertise for many ministries of health, especially given their many divergent priorities. On the other hand, external partners who traditionally have the resources to conduct such research may not always understand the primary operational challenges confronting ministries of health and may have research interests disparate from them. The resulting studies and their results may not be immediately applicable to the countries’ programmatic needs. It is the responsibility of both parties to ensure that relevant research of the highest quality is conducted. The Global Immunization Vision and Strategy suggests that: “Where appropriate, perform operations research and evaluation of ‘what works’ to improve the delivery of immunization and to make systems more effective, efficient and equitable in order to improve immunization coverage.”⁴ Additional funds for immunization and health services research have recently been made available through the GAVI Alliance’s Health System Strengthening efforts and potentially through other mechanisms to support much-needed operational research. Ministries of health and external partners must work together to ensure that these additional resources are used to identify and test innovative approaches and strategies to meet the priority programmatic needs and ensure that the operations research is of “high quality”.

Finally, the paucity of studies found in our review suggests a need for additional mechanisms to disseminate findings of operations research conducted at the local level. The lack of studies may be due in part to the number of accessible journals or their requirements for submission. We support efforts to ensure that systems are in place to foster communication between ministries of health on what has been tried, what works and does not work. The failure to provide a forum to disseminate the operations research results to other countries is a missed opportunity. ■