How to and how not to develop a theory of change to evaluate a complex intervention: reflections on an experience in the Democratic Republic of Congo

Abstract

Theories of change (ToCs) describe how interventions can bring about long-term outcomes through a logical sequence of intermediate outcomes and have been used to design and measure the impact of public health programmes in several countries. In recognition of their capacity to provide a framework for monitoring and evaluation, they are being increasingly employed in the development sector. The construction of a ToC typically occurs through a consultative process, requiring stakeholders to reflect on how their programmes can bring about change. ToCs help make explicit any underlying assumptions, acknowledge the role of context and provide evidence to justify the chain of causal pathways. However, while much literature exists on how to develop a ToC with respect to interventions in theory, there is comparatively little reflection on applying it in practice to complex interventions in the health sector. This paper describes the initial process of developing a ToC to inform the design of an evaluation of a complex intervention aiming to improve government payments to health workers in the Democratic Republic of Congo. Lessons learnt include: the need for the ToC to understand how the intervention produces effects on the wider system and having broad stakeholder engagement at the outset to maximise chances of the intervention’s success and ensure ownership. Power relationships between stakeholders may also affect the ToC discourse but can be minimised by having an independent facilitator. We hope these insights are of use to other global public health practitioners using this approach to evaluate complex interventions.

Key questions.

What is already known about this topic?

• Theories of change (ToCs) help to articulate the change process within complex interventions.

• Their use has increased exponentially in the development sector.

What are the new findings?

• A number of practical issues to ensure the successful formulation of a ToC were identified and included:

  • The need to consider how a complex intervention may interact with the wider system rather than being considered only in the context of one sector.

  • The importance of identifying and ensuring adequate input from all of the relevant stakeholders.

  • The need to involve all stakeholders in the conception of the ToC to encourage ownership.

  • The role of the facilitator as an objective broker of power relationships between stakeholders.

Recommendations for policy

• These practical issues may help partners to use the ToC approach to its full potential, creating space for critical reflection rather than being an illusory process.

Introduction

Complex interventions are commonly defined in the literature as interventions that comprise multiple components acting both independently and in conjunction with one another.^{1 2} Other characteristics contributing to their complexity include: the number and difficulty of behaviours required by those delivering or receiving the intervention, number and variability of outcomes and the degree of flexibility permitted within the intervention.³ A theory of change (ToC) approach can be an effective way to evaluate such interventions by taking into account implementation aspects, mechanisms of impact and the effects of context.⁴ When a complex health systems intervention is being evaluated, it is necessary to understand how the intervention relates to and interacts with components of the system to produce an effect. In applying this ‘systems thinking’ approach, multiple points of view or perspectives need to be sought.⁵

Originally developed by Weiss,⁶ ToCs articulate the change process within interventions and describe the sequence of events linking intervention activities to their long-term outcomes.⁷ They make explicit the conditions and assumptions required to enable change^{8 9} and acknowledge the role of context in influencing the process. Diagrams are often used to depict a ToC as most complex interventions consist of elements interacting in a non-linear fashion, with indirect causal pathways and feedback loops. This is in contrast to logic models and logical frameworks, which tend to be more rigid and linear in outlining the inputs, processes, outputs and outcomes of an intervention.⁴ ToCs are also dynamic and should be modified throughout the evaluation to reflect any diversions during implementation from the original theory. As such, it is recognised that the process of conceiving an intervention’s ToC is an ongoing and iterative process, requiring regular review throughout the evaluation of the intervention. ToCs are typically developed in collaboration with stakeholders in order to build consensus on the change process. Their use has increased exponentially in the development sector⁴; a recent systematic review identified 62 studies employing ToCs to evaluate public health interventions.¹⁰ However, there is comparatively little reflection on the process of using this method to design an evaluation.

This article describes the experience of using stakeholder workshops to construct an initial ToC that will go on to inform the evaluation of an intervention aiming to improve health worker payment and motivation in the Democratic Republic of Congo (DRC). It provides a range of lessons learnt from this process. A description of how the ToC is subsequently used and updated through a process evaluation will be the subject of a separate research article.

Background

The DRC is a fragile and conflict-affected state with several health system challenges, including a dysfunctional public sector wage system.^{11 12} Most health workers do not receive a salary,¹¹ which impacts on motivation, the staffing of facilities and therefore the delivery of healthcare. The poor remuneration of health workers stems from a government failure to update the payroll, a lack of sufficient resources and governance to address the problem and corruption, which has allowed the proliferation of unofficial appointments.

In 2013, the Department for International Development (DFID) started funding a 5-year health systems strengthening programme in DRC called ASSP (Accès aux Soins de Santé Primaires/ Access to Primary Healthcare), implemented through Interchurch Medical Assistance (IMA Worldhealth), which included an intervention to facilitate the payment of government salaries to health workers. This intervention involved: the establishment of an electronic open source Human Resources Information System (iHRIS) through a census of health workers and the definition of optimal staffing standards using the Workload Indicator of Staffing Need procedure, which together were aimed at ensuring sufficient workers were in place and that they were appropriately paid.

To date, similar interventions using iHRIS to record census data have been reported in other countries in the form of case studies.^{13 14} In Sierra Leone, strong political will to improve governance was identified as a key condition for successfully ensuring the integrity of the health sector payroll.¹⁴ However, these evaluations lacked a theoretical framework from the outset and focused on measuring outcomes rather than processes involved in implementing the intervention. Consequently, there is limited understanding of how to replicate such interventions effectively in different settings.^15–17 Given this, a ToC approach was employed to clarify how the intervention in the DRC would translate into its intended effects, thus informing the design of an evaluation of the intervention.

Developing the ToC

A workshop was convened to develop the initial ToC with principal investigator RM as facilitator using guidance from previous training on constructing ToCs at the Wellcome Trust.¹⁸ Stakeholders were identified through discussions with implementing partners and DFID. This workshop lasted 3.5 hours, with representation from IMA, DFID and Intrahealth as well as two staff conducting operational research on the programme.

During the workshop, RM described the ToC approach using examples and then asked stakeholders to undertake an exercise. The remainder of the workshop required stakeholders to collaboratively construct the ToC for the intervention of interest. Ground rules were established, requiring participants to: show respect for others’ opinions, put aside personal agendas, be open-minded and agree to decisions made on the consensus view.

As human resources and health financing were two of the recognised building blocks of the health system being targeted by the intervention,¹⁹ the ToC approach did not exclusively focus on the intervention but rather aimed to unpack the effects the intervention would have on the health system or the expected mechanisms of programme effect (how it works). On identifying the long-term goal of the intervention, stakeholders then worked back from this through to the earliest changes that needed to occur using the process of ‘backwards mapping’. They were asked to populate coloured Post-it notes with different colours representing: the long-term outcome, preconditions or intermediate outcomes needed to achieve the final outcome, activities needed to move from one outcome to the next, assumptions (those conditions in which the intervention will take place which must hold true for the ToC to be realised), rationale for each link in the causal pathway and indicators (to evaluate whether the intermediate and final outcomes have been achieved). The facilitator then discussed the Post-it notes with the group and added those agreed on to a flip chart in order to map out the ToC.

Not all of the key stakeholders were present at this workshop; although Ministry of Health officials were invited, they did not attend. It also emerged during the workshop that government stakeholders from different ministries would need to be involved. Therefore, another meeting was organised that lasted 1 hour and included government representatives from the Ministries of Health, Public Service, Finance and Budget, as well as DFID, IMA and Intrahealth. All those present were asked to review a refined diagram of the ToC constructed on the back of the discussions from the first workshop.

Interviews were also held with individual stakeholders on the ToC both prior to and after implementation of the intervention, and each respondent was asked to review and comment on the final ToC. Interviewed stakeholders included those involved in the workshops, health workers who would be affected by the intervention, government officials at the provincial level and other donors in the health sector.

Results

The resulting ToC is given in figure 1. As well as focusing on the implementation steps, the ToC describes the hypothesised mechanisms of change occurring within the health and wider public system. In addition to the two main programme activities, the ToC outlines the crucial input of government in endorsing staffing standards and using iHRIS and committing the necessary resources to funding salaries. The intervention is expected to improve the availability and distribution of health workers, increase their motivation to deliver quality care and reduce user fees (a key source of their revenue), making services more accessible to the population.

Figure 1.

Initial theory of change for a complex intervention to improve the payment of government salaries to health workers in the DRC. ASSP, Accès aux Soins de Santé Primaires/Access to Primary Healthcare; CODESAs, Comité de Développement de l’aire de Santé/Community health committees; DRC, Democratic Republic of Congo; DFID, Department for International Development; FP, Fonction Publique/Ministry of Public Service; HR, Human Resources; iHRIS, Human Resources Information System; IMA, Interchurch Medical Assistance; MoH, Ministry of Health; WISN, Workload Indicator of Staffing Need.

Reflections on the process and lessons learnt are described below.

Reflections

What worked well

In the first workshop, participants who were not well versed with ToCs found it particularly useful to be given a simple example of a ToC to help familiarise them with the terminology and understand the definitions and distinctions between assumptions and preconditions. The example provided related to an intervention to deliver measles vaccinations to children under 5 years of age, which had been devised by RM. This worked well because it was a relatively simple intervention that respondents were familiar with. While it may have been helpful to have guided participants through a more complex health systems strengthening example, which would have also emphasised the importance of applying systems thinking to the conception of the ToC, this had to be balanced against using a short and simple example that illustrated the concepts fundamental to developing a ToC.

Based on their roles in the intervention, stakeholders held very different views on what the long-term goal of the intervention should be. For example, DFID believed that the overall goal of paying health workers was to improve the quality of healthcare, while IMA thought it would remove incentives to charge patients high user fees. Through facilitation, the group conceded that both of these were expected outcomes of remunerating workers and aligned with the overall goal of increasing access to healthcare. Therefore, a positive outcome was that the ToC process brought respondents together to reach consensus on goals as well as widen their perspective on the different effects of the intervention. It also meant that stakeholders spent time unpacking the change process, reflecting on the connections between activities, outcomes and impacts, thereby revealing linkages that had previously been overlooked.

What did not work well

Failure to implicate all of the relevant government ministries with significant influence over civil service reform—namely the Ministries of Budget, Finance and Public Service—from the outset meant the intervention was initially viewed within the confines of the health sector; adopting this narrow perspective meant that the wider public institutional system within which this intervention was situated was being ignored. Ideally, a ToC would also be used to inform the design of an intervention, and the system that the intervention needs to change should be mapped first, in order to identify potential leverage points where the intervention can bring about change. In this case, however, the technical components of the intervention had already been agreed prior to the development of the ToC, so the ToC was used to guide the evaluation.

To enable sufficient government engagement, another meeting was held and integrated into the ASSP project’s quarterly review meeting. However, a major limitation was that only 1 hour was allotted to the ToC in the agenda, during which the ToC developed during the first workshop was presented. Government stakeholders expressed little desire to comment or modify the ToC perhaps because they felt unable to challenge the existing theory that had been put forward. It is also plausible that the government did not fully understand ToCs as time was insufficient to go through a worked example. Another possibility is that the ToC was perceived to be more donor driven and/or did not reflect the reality of the DRC. Consequently, the government may have felt less ownership of the process but were also unwilling to go into any depth on the specific activities that they needed to lead on to enable success, such as how they would use the information yielded by the census to improve the payroll. Overall, this resulted in poor contribution of the relevant ministries to the conception of the ToC. To mitigate this, two sets of follow-up interviews with key individuals were later held on the ToC for the intervention.

In the first workshop, implementing partners were reluctant to define indicators to monitor the intervention, as they felt failure to achieve targets associated with the indicators would be an indictment of their performance. However, they became more receptive once DFID acknowledged that the success of the intervention did not solely depend on their performance, as articulated by some of the assumptions. The idea that everyone has a role with no one group being responsible overall could have been better communicated by the facilitator at the beginning.

The opportunity to identify which stakeholders were responsible for progressing various elements of the ToC was missed when developing the ToC; making this more explicit would have had the advantage of clarifying responsibilities and strengthening accountability. However, this had to be traded against stakeholders perceiving the ToC as a way of controlling and monitoring their actions, rather than a tool to help test the hypotheses and assumptions of the intervention.

The expertise required to implement some of the technical activities for the intervention was to come from abroad. Therefore, at the outset, there were questions around the adequacy of the intervention design and applicability of the intervention to the DRC context. In addition, some of the articulated activities were very vague, for example, those that required ‘advocacy with government’. Yet, the ToC helped to make these areas of uncertainty more explicit, thereby providing a focus for further work. Given the composition and interests of stakeholders involved in constructing the ToC during the first workshop, there was a degree of optimism bias that the intervention would be successful. This sometimes made it challenging to identify any potential negative or unintended consequences of the intervention. Power dynamics were also evident between stakeholders and possibly attributable to their differing incentives. For example, as implementing partners are reliant on donors for funding, this may have influenced their ability to maintain an independent perspective and position. Similarly, the Ministry of Health may have felt inhibited to speak out in the presence of the more powerful ministries involved in pay reform.

There was a lack of local engagement in both of the workshops; the views of health workers who would be the ultimate beneficiaries of the intervention were not elicited. This had the drawback of not giving them a voice or considering their needs; as a result, follow-up interviews on the ToC included health workers in the sample.

During follow-up interviews, a proposed modification to the intervention to improve its chance of success included working with other donors on the DRC’s retirement policy. However, implementing partners and DFID were reluctant to consider this, perceiving it to be ‘scope creep’ that would also require far more resources.

Lessons learnt

• A systems thinking approach may require looking beyond individual sectors. In this case, the process of developing the ToC was valuable in enabling consensus on the effects the intervention would have on the health system but should have given more consideration to the wider institutional system at the outset. Systems mapping is a tool that can be used in conjunction with ToCs to reach a deeper understanding of the system and help identify potential leverage points.

• Identify all key stakeholders from the outset. Stakeholders are identified as those with an interest in the intervention, are affected by the intervention or those who may have an active or passive influence over decision-making and implementation processes.²⁰ It is necessary to be as inclusive as possible at the beginning, recognising the importance of a range of perspectives in understanding the theoretical basis for the intervention.²¹

• Invest time in ensuring a similar level of understanding of ToCs among stakeholders. Often, stakeholders will not have heard of ToCs or have different ideas of what a ToC is,²² therefore starting the workshop with a presentation on ToCs and some illustrative examples can ensure everyone has a similar level of understanding at the start. This will also ensure everyone feels confident enough to contribute to the development of the ToC.

• Prepare to manage stakeholder dynamics. In this case, unequal aid relations existing between donors and implementing partners, as well as between ministries may have affected the ToC discourse. Hence, having an independent facilitator can help ensure a degree of objectivity and mitigate any power imbalances. Other approaches may include limiting the participants in the workshop and stratifying groups to ensure open and honest contributions.²³ However, a multistakeholder workshop has the advantage of ensuring the sharing of different perspectives, clarifying roles and responsibilities, and lays the basis for collaboration.²⁴

• Avoid presenting a ready-made ToC. Give space to all stakeholders to develop the intervention theory themselves. Although more time-intensive, particularly for those not familiar with the intervention, it is important in overcoming the phenomenon of ‘group think’, which can occur with those involved in conceiving the intervention. It will also ensure greater ownership of the end product by those with any influence over implementation.

Conclusion

The ToC process had utility in bringing respondents together to reach consensus on the mechanisms of effects of the intervention and its desired outcomes. It also demonstrated the importance of applying a systems thinking approach that helped in identifying and engaging all stakeholders who could influence the success of the intervention. The role of the facilitator was key to: ensuring stakeholders have a clear understanding of ToCs, mitigating any power imbalances and encouraging a critical, honest and reflective approach.

Nonetheless, particularly with politically sensitive interventions like this, the process was not straightforward. It is hoped that the insights here shed light on what to do and what not to do and will encourage others to share their experiences to guide those using ToCs in their research.