Addressing the tensions and complexities involved in commissioning and undertaking implementation research in low- and middle-income countries

Summary box.

• Rapid scale-up of new policies and guidelines, in the context of weak health systems in low/middle-income countries (LMIC), has led to greater interest and funding for implementation research.

• Implementation research in LMICs is often commissioned by institutions from high-income countries but increasingly undertaken by LMIC-based research institutions.

• Commissioned implementation research to evaluate large-scale, donor-funded health interventions in LMICs may hold tensions with respect to the interests of the researchers, the commissioning agency, implementers and the country government.

• We propose key questions that could help researchers navigate and minimise the potential conflicts of commissioned implementation research in an LMIC setting.

Against a background of rapid scale-up of new policies and guidelines in the context of weak health systems in low/middle-income countries (LMIC), implementation research is especially important for understanding the ‘evidence-implementation’ gap. Implementation research investigates the various factors that affect how a new health policy or intervention may be implemented in usual practice settings and the contextual factors that affect implementation at scale.^1–4 A wide range of qualitative and quantitative methods can be used in implementation research including: pragmatic trials, quality improvement studies, participatory action research and mixed methods evaluation studies where both quantitative and qualitative methods of data collection and analysis are used in the same study.¹

There has been an increase in funder requests for research proposals that document and evaluate implementation strategies and impacts of large-scale health interventions aimed at supporting the delivery of health services, programmes and policies.⁵ The concomitant emergence of ‘global health’ as a field within North American and European universities (referred to here as the ‘Global North’) has led to a plethora of institutions who commission, undertake and collaborate on implementation research within LMIC settings (referred to here as the ‘Global South’), sometimes as part of countries’ overseas development assistance (ODA).^{6 7}

Recently, there is a welcome, although gradual shift towards greater commissioning of research institutions in the Global South to undertake research on programmes implemented in and/or issues affecting the Global South.⁸ However, the commissioning agencies are commonly situated in the Global North. In this opinion piece, we reflect critically on the inherent tensions involved with undertaking such commissioned implementation research and the potential for North-South power differentials to add to this complexity.^{7 9} Many of the dilemmas raised may well apply to other types of health research. Our perspectives emerge from our personal and collective experiences as health systems researchers working in LMIC settings.

The political context of commissioning and undertaking implementation research

In many LMICs, health sectors are typically dependent on ODA for the funding and the evaluation of health programmes, which often take the form of large-scale health service and programme implementation, geared towards addressing priority health problems. Research agencies may be contracted to undertake evaluation research, be it by appointment or via a competitive bidding process.¹⁰ Undertaking such commissioned implementation research has inherent tensions stemming from differences in objectives (explicit and implicit) between the different stakeholders involved; including commissioning agencies, funders of the implementation programme, researchers and the government of the host country.

In an LMIC context, often with weak health systems and governance structures, there may be a dependence on donor funding for implementation, scale-up and sustained delivery of health services and programmes. Ethical and political dilemmas of undertaking commissioned research have been acknowledged by others.^{11 12} Mumtaz et al¹² have argued that projects commonly have unarticulated goals alongside explicit health outcomes which influence how and which research findings are shared, including: ‘maintaining positive relationships between donors and governments, keeping money flowing, and maintaining the appearance of success.’

We propose two key questions we feel can help researchers be aware of, anticipate and potentially minimise the complexities and pitfalls associated with commissioned implementation research,¹¹ especially in a setting where there may be a North-South power differential: (1) Who is commissioning the research and how can the independence and integrity of the research process be maintained? (2) What is the purpose of the research and how will the findings be used? Boxes 1 and 2 contain illustrative vignettes reflecting lessons we have learnt as health systems researchers from an LMIC setting, who through competitive processes, have undertaken independent implementation research of complex health system interventions.

Box 1. Who is commissioning the research and how can the independence and integrity of the research process be maintained?

A multilateral agency has advocated for countries to scale up outreach services for child immunisations in ‘hard to reach’ areas. The agency has supported several countries in sub-Saharan Africa to implement this intervention and they have now commissioned an independent evaluation of the intervention. Following a competitive process, a research institution from the Global South is awarded the research grant.

During the planning meeting with the commissioning agency the researchers discuss their plans for undertaking the evaluation. The agency informs them that they would like their country staff to accompany the researchers during their fieldwork, including sitting in on interviews with stakeholders. They argue that this independent evaluation could be a learning opportunity for their staff. The commissioned researchers acknowledge that the country agency staff have a role to play in facilitating the logistics of the in-country fieldwork (such as assisting with setting up interview times and venues with local stakeholders and providing relevant documentation for review). However, the researchers feel that the presence of the agency staff in interviews represents a conflict of interest, could inhibit honest dialogue with those interviewed and thereby poses a threat to the independence of the research and thus to the integrity of the data collected. Following discussion, the commissioning agency agrees to the request for them not to be present during interviews.

The researchers also considered the ethical principle of non-malfeasance, to ensure the research does not cause harm. The commissioning agency was also supporting the salaries of some stakeholders being interviewed as part of the research. These participants may be particularly vulnerable to negative consequences (perceived or real) should they be open in interviews about the shortcomings of the programme being evaluated. The researchers have to ensure that they sufficiently anonymise the data and research report, so as to protect the identity of participants.

Box 2. What is the purpose of the research and how will the findings be used?

A high-income country government has funded the scale-up of ‘universal test and treat for HIV (UTT)’ in six countries in sub-Saharan Africa. The grant period is 5 years and they have budgeted for an ‘end of grant’ evaluation in the final year. A university-based research agency from the Global South is awarded the research grant to undertake a mixed methods evaluation of progress with implementation of UTT in the focus countries. There are certain restrictions in the contract. First, the research questions were predetermined by the funder and, second, the researchers are required to obtain permission from the funder before any publication of the findings.

The researchers complete their fieldwork and submit the report to the funder. One of the main findings from the research is that the weak health systems in the countries are hindering scale-up of the programme, especially weak supply chain systems which result in stock-outs of HIV test kits and antiretrovirals. Since the funder plans to use the evaluation to lobby their constituents for continued funding, they, together with the country government, request that the researchers write short case studies highlighting areas of success. They also request that researchers not publish the findings related to health systems weaknesses.

The researchers, while cognisant of the risks to future funding for the programme, argue that publication of the health system-related findings is as important for reasons of scientific integrity, and could contribute important evidence for future decision-making, planning and resource allocation for HIV services in low-income, weak health systems. A discussion ensues between the researchers and the commissioning agency and a middle ground is reached where the health systems findings are published in a peer-reviewed journal following revisions to the satisfaction of the agency.

As illustrated here, in an environment of high donor dependence and short-term donor commitments, stakeholders, especially country governments, may be particularly interested in findings that support ongoing investment in programmes. Commissioned researchers therefore need to be cognisant of the purpose of the research and initiate discussions around the implications of the possible findings at the outset and there should be conflict resolution processes in place.

Who is commissioning the research and how can the independence and integrity of the research process be maintained?

Implementation research can be commissioned by a variety of organisations including national governments, private sector companies, implementing agencies, bilateral and multilateral agencies.¹⁰ It is important for researchers to reflect on who the commissioning agency is and on the potential complications this may have for the independence and integrity of the research. For instance, it is not uncommon for implementation research to be commissioned by agencies which are themselves both funding and implementing health programmes being evaluated. These agencies may also themselves be involved with developing global guidelines for the interventions they are having evaluated. Understandably, these agencies may therefore have a particular interest in showing a positive impact or other results that would support their objectives. It may be difficult for the commissioned researchers to establish their independence at an early stage, by for instance asking critical questions that may further shape the research scope and by ensuring the research methodology is sufficiently independent. In box 1 our vignette illustrates the dilemma of ensuring independence in data collection methods, in a situation where the boundaries between the key stakeholders were fluid.

We illustrate in box 1 the importance of establishing independence early in the commissioning processes, to uphold research integrity. Important considerations are whether the commissioning agency is also the funding or implementing agency and thus stands to potentially gain or lose future funding for the programme being researched, based on the direction of the findings, and whether the interviewees are directly or indirectly funded by the commissioning agency.

Most agencies commissioning implementation research strive to appoint independent researchers to carry out the research. However, as illustrated, there are instances in which members of the funding or programme implementation team are involved in designing the research, selecting research sites and even assisting with data analysis and interpretation. This situation greatly increases the influence of the commissioning agency in the research process, thus potentially influencing the outcomes. Where funders are involved in this way, we suggest that the commissioning agency and the researchers work towards clear guidelines regarding their respective roles and responsibilities. This should include discussion of the extent to which the researchers have the final autonomy for the collection, interpretation and reporting of findings; and what mechanisms will be put in place to ensure the independence and integrity of the research process.

What is the purpose of the research and how will the findings be used?

The commissioning of implementation research can serve multiple purposes: the implementing agency (government, non-governmental organisation or international implementing partner) may be required to report to a donor agency about progress with implementation or impact, as a requirement for continued funding; or an assessment of a pilot phase may be required before a decision can be made about moving to scale¹²; or experience with implementing an intervention may need to be documented in order to strengthen implementation or uptake.¹³ Agencies also commonly commission ‘end of programme’ evaluations aimed at providing evidence of impact and cost-effectiveness in order to motivate host countries to take over funding of a programme.

Dissemination of implementation research is influenced by the perceived purposes of the research and the types of outputs expected by different stakeholders. Researchers may want to publish peer-reviewed articles, funders want a report and impact analysis, governments and partners often want presentations and briefs to accompany a report. A clear understanding of expected outputs is therefore needed when undertaking implementation research, including the way in which different stakeholders can use the research findings.¹⁴

There are important ethical considerations in the dissemination of research findings: overt or covert restrictions on the dissemination of research findings raise dilemmas regarding data ownership and prevent the opportunity to learn from the research findings to improve implementation.¹² Furthermore, if a government or multilateral agency funds research with public money, it could be argued that there is an ethical responsibility to make the findings publicly available regardless of the implications.

It is important for researchers to understand the types of decisions that may be taken as a consequence of research findings. For example, what might be at stake for the commissioning agency, the implementers or the government if the research findings highlight challenges with implementation, health systems constraints to implementation at scale, or no success or impact? Researchers also have obligations to ensure that their team have the necessary experience and competencies to carry out high-quality research that is defensible in the face of criticism or alternative interpretations.

Understanding the purpose of the research is important for the commissioned researchers so that they can ask the most appropriate questions and can anticipate the consequences of various research findings. In box 2 we illustrate the inherent tensions arising from different expectations and views of the research purpose and how this may play out in disagreement about the research scope, interpretations of findings and dissemination of study findings.

Recommendations for implementation researchers

Undertaking commissioned implementation research is likely to come with tensions relating to the differing interests of stakeholders including the commissioning agency, implementation agencies, host country governments and the commissioned researchers. Competing priorities among stakeholders may give rise to contestation at various stages of the commissioned research, including the conception stage, implementation stage, interpretation of findings and the dissemination and utilisation of findings. Commissioned researchers from LMIC settings may feel (real or perceived) disempowerment in navigating this complex terrain¹⁵ and this article attempts to describe some of these complexities.

In a critical reflection of the limited impact of global health development efforts on maternal health in LMICs, Mumtaz¹² and colleagues argued that we need to learn from ‘successes and failures’, but that this would require a kind of collaboration which is currently absent (due to political, social and financial pressures in LMICs), where there is consensus among all stakeholders on the need for evidence of what works and what does not work, regardless of the implications.

In the absence of such consensus, we are advocating for researchers to be cognisant of the complexity of undertaking commissioned implementation research in LMIC settings. We propose that asking key questions about the relationships among stakeholders involved in the research can help clarify the potential conflicts that may emerge. The focus of such discussions should be how the research can be conducted in a way that ensures the independence and integrity of the research process, how the objectives and needs of various stakeholders will be considered and what conflict resolution mechanisms are required. This may be particularly challenging given the economic reality that research institutions from the Global South experience, resulting in pressure to sacrifice negotiating ability in order not to jeopardise much needed research funding.⁶

Beyond supporting their own researchers in these negotiations, institutions from the Global South could also link together to support each other and to share strategies and modified approaches to ensure fair research contracting and commissioning^{7 9} drawing on existing initiatives such as the Council on Health Research for Development which provides guidance on fair research contracting. Additional recommendations include strengthening accountability structures governing global health practice, the establishment of national ombudsman or mediating agencies and an international evaluation registry, similar to a clinical trial registry, to increase transparency and reduce selective reporting of programme evaluations.¹²

The political context of commissioning implementation research is a reality and the questions posed in this article could be useful for researchers when considering undertaking commissioned research and for engagements with commissioning agencies. These questions could also spark open discussion between these stakeholders towards the primary goal of strengthening health systems.