Time for innovative dialogue on health systems research

A forthcoming report by the World Health Organization (WHO) on evidence for policy is already fuelling passionate debate, especially among scientists. Opinions are divided: some scientists believe strongly in the systematic assessment of evidence on the effectiveness of health system interventions and reforms;¹ others are skeptical.^2,3

This is not just another academic debate. Health policy is a noisy field involving many stakeholders. Health reform approaches that seem promising in practice are assessed critically by institutional researchers, whose methodological dictums are sceptically received by field practitioners. In a recent Bulletin editorial, for example, Fretheim et al. reported the results of their Cochrane review of performance-based financing studies, which yielded few studies meeting Cochrane criteria. When the authors concluded that little could be said about performance-based financing,^4,5 advocates of this approach responded with an avalanche of critical commentaries.⁶

These debates are typical, necessary and energizing. They hail a new era in the worldwide scrutiny of health policy and its knowledge architecture. As legislators, researchers, knowledge managers and contributors to health reform, we welcome the deepening interest in the intricacies of health reform and acknowledge the global relevance of an ambitious new knowledge agenda.

A balanced dialogue between different knowledge holders – researchers, practitioners and policy-makers – is, in our view, a conditio sine qua non for successful health policies. It concerns us all to consolidate the body of evidence surrounding promising approaches, including data on their possible pitfalls and side-effects. For such a dialogue to take off and become truly global, the following prerequisites should be fulfilled.

All contributors should realize that the knowledge agenda for successful reforms extends beyond their own niches.⁷ Serious knowledge programmes for improving health systems in low- and middle-income countries go well past merely scrutinizing evidence, and certainly outcome evidence. Our own experience has taught us that most schemes anywhere fail because of poor design, insufficient funding, weak governance and muddled implementation. These dimensions of the health system development process warrant proper attention from researchers. In addition to the hierarchy for the “strength of the evidence”, there should also be a hierarchy for practical relevance.

We should revisit current evaluation methods or at least their interpretation. The broad transformative character of many health system reforms, such as performance-based financing, poses challenges for evaluation methodology.⁸ Promising features of health reforms can stem from dimensions not amenable to randomization, such as professional culture and accountability; research methods limited to one or two outcome parameters are too reductionist.

Scientists should always strive for rigour in evaluating reform interventions. The Cochrane standards, for instance, provide useful guidance for better design and reporting of primary studies. However, health reform is not a simple intervention: context matters a lot and contents resist homogenization. Health sector reform is often about modifying institutional arrangements, which differ considerably across countries and settings. Hence, the findings in one country will frequently have limited applicability in another. Researchers seeking to conduct impact studies or synthesize their findings should fully familiarize themselves with local contexts, stakeholder development dynamics and implementation strategies in the field.

We should be aware that nesting an impact evaluation within a comprehensive health reform effort is not a neutral operation. It affects the policy process and may exhaust or undermine its momentum. Ironically, the research may produce systemic outcomes not captured by the randomized design. These should be meticulously documented.

Finally, we urgently need to think about where to conduct the interactions on the knowledge agenda for better health policies. Engaging environments are required for open and productive communication across professional boundaries. A new research dialogue cannot be staged exclusively in scientific journals. Communities of practice, such as those affiliated with Harmonization for Health in Africa, gather large numbers of people with fair distribution of input and experiential knowledge. While cohesive and animated, they build trust and allow for critical collaborative dynamics, desperately needed by health systems research.⁹

In the end, any dialogue in a tortuous policy process also requires realism. At the country level, health decision-makers face huge uncertainties. The windows of opportunity are frequently unrelated to the rigour of any evidence available. The recent nationwide roll-out of a new treatment approach for pregnant mothers infected with the human immunodeficiency virus (HIV) in Malawi was based on expert opinion (and advocacy), not a randomized controlled trial.¹⁰ Similarly, the Burundian government may not wish to wait for a four-year study to end if it senses promise in a strategy that can meet pressing health needs today. All the more reason to innovate researchers’ dialogue with parties in the thick of it.