Verbal Autopsy with Participatory Action Research (VAPAR) programme in Mpumalanga, South Africa: protocol for evaluation

Abstract

Introduction

There is a growing recognition of the importance of developing learning health systems which can engage all stakeholders in cycles of evidence generation, reflection, action and learning from action to deal with adaptive problems. There is however limited evaluative evidence of approaches to developing or strengthening such systems, particularly in low-income and middle-income settings. In this protocol, we aim to contribute to developing and sharing knowledge on models of building collaborative learning platforms through our evaluation of the Verbal Autopsy with Participatory Action Research (VAPAR) programme.

Methods and analysis

The evaluation takes a participatory approach, focussed on joint learning on whether and how VAPAR contributes to its aims, and what can be learnt for this and similar settings. A realist-informed theory of change was developed by the research team as part of a broader collaboration with other stakeholders. The evaluation will draw on a wide variety of perspectives and data, including programme data and secondary data. This will be supplemented by in-depth interviews and workshops at the end of each cycle to probe the different domains, understand changes to the positions of different actors within the local health system and feedback into improved learning and action in the next cycle. Quantitative data such as verbal autopsy will be analysed for significant trends in health indicators for different population groups. However, the bulk of the data will be qualitative and will be analysed thematically.

Ethics and dissemination

Ethics in participatory approaches include a careful focus on the power relationships within the group, such that all groups are given voice and influence, in addition to the usual considerations of informed participation. Within the programme, we will focus on reflexivity, relationship building, two-way learning and learning from failure to reduce power imbalances and mitigate against a blame culture. Local engagement and change will be prioritised in dissemination.

Strengths and limitations of this study.

• It combines realist approaches with participatory action research and allows for testing and refinement over several cycles.

• Risks related to power imbalances and insider/outsider tensions are acknowledged, with mitigating strategies planned.

• It aims to add to the limited literature on collaborative learning platforms to support learning health systems in low-income and middle-income settings.

• The study will also build the field of evaluation of participatory research at multiple levels of the local health system.

• One limitation is that results will be specific to this site, however broader engagement is planned to allow for sharing of lessons with other learning sites using related models.

Introduction

Background on the VAPAR programme

The Verbal Autopsy with Participatory Action Research (VAPAR) programme started in 2017 in Mpumalanga province, South Africa, as a partnership of local and international researchers, community members and health system stakeholders. Its aim is to embed a system of knowledge production and exchange for health systems strengthening in order to improve services and outcomes for vulnerable group’s health locally and with the potential, if successful, for wider learning, uptake and sustainability (www.vapar.org).

In VAPAR, data from verbal autopsy (VA) and participatory action research (PAR) is combined in a series of reflection-and-action cycles based on continuous quality improvement for health systems strengthening, engaging relevant stakeholders at different levels of the health system (figure 1). The programme consists of three learning-and-action cycles over 2017 to 2022, with each cycle of the VAPAR programme including four stages: observe, analyse, plan and act.¹

Figure 1.

Verbal Autopsy with Participatory Action Research action learning cycle.

The VA component incorporates new WHO indicators developed with the Mpumalanga Department of Health during pilot work in 2015/2016 on social and health system factors contributing to the ‘circumstances of mortality’¹ within VA data gathered through the Medical Research Council (MRC)/Wits Agincourt Unit's Health and Socio-Demographic Surveillance System (HDSS) in Mpumalanga. These VA outputs are shared during PAR with village-based groups, which identify priority topics, analyse root causes and the impact of the problem, identify stakeholders and plan action to address these with key stakeholder groups. During the pilot phase of the programme in 2016, the areas of under-5 mortality and HIV infection were nominated by researchers and the Department of Health; however in the main phase communities nominated access to clean water, alongside alcohol and drug abuse as key local health priorities.^2–4 The last two topics were carried through by communities into the main phase in 2017. Data from VA and PAR have then been interpreted with district and provincial stakeholders (within the health sector but also beyond, as relevant) in order to reach common understanding on problems and root causes, leading to actionable agendas and promoting learning from that action in ongoing processes.

The research is informed by established post-positivist paradigms asserting that all truths are partial. The work is thus rooted in and draws from critical,⁵ constructivist⁶ and participatory/cooperative traditions.⁷ Through paradigmatic interweaving, we seek to deepen understanding in and of health systems as complex adaptive systems and social constructions⁸ located in and continually (re)shaped by wider social, political and historical contexts, and cooperative enquiry as political participation in collaborative practical knowing, action and transformation. Drawn together, the key underlying assumption is that practical, experiential knowledge that is co-constructed, self-reflective and embedded in complex, adaptive social and health systems will support and inform the organisation and delivery of public goods that are equity-oriented and people-centred. The beneficiaries are intended to be people and practitioners in resource-constrained systems, collectively possessing rights and responsibilities for health, healthcare and wider public services.

Context

VAPAR is situated in Mpumalanga, South Africa, a rural province of 4.6 million people in the northeast, bordering Swaziland and Mozambique. Conditions in poor and rural villages are comparable with many other settings in the region: there is limited piped water, rudimentary sanitation, underdeveloped roads, unaffordable electricity and high unemployment.⁹ The burden of HIV in South Africa is high and highly unequal. Prevalence in black populations is 40 to 50 times that of white and in adolescents, risks are eight times higher in female adolescents than males.¹⁰ Age-adjusted HIV prevalence in the study area is 26% in women and 19% in men.¹¹

In spite of entrenched social and health inequities, the post-apartheid policy context in South Africa is progressive and inclusive. There is a constitutional commitment to the right to health and community participation for primary care,¹² and National Health Insurance was launched in 2012 as a clear commitment to Universal Health Coverage.¹³ Despite a ‘near-ideal’ policy context, there is chronic underinvestment in public services. This has resulted in human resource crises, corruption, poor stewardship and deteriorating infrastructure — and deep disconnects between policy and implementation as a result.¹⁴ The health system also faces a complex ‘quadruple’ burden of socially patterned disease comprising: chronic infectious diseases (characterised by HIV/AIDS and tuberculosis), non-communicable conditions, maternal and child mortality and mortality owing to injury and violence.^{15 16}

Purposes of evaluation

In this paper, we lay out the approach which will be taken by our evaluation, which will be participatory among our key constituencies and will aim to understand whether and how VAPAR contributes to its aims, and what can be learnt for this and similar settings. We present the theory of change of the programme, how it was developed and is evolving and how it will be tracked using mixed methods. We discuss our positionality and some of the risks and ethical issues arising.

In addition to informing the development of the programme, the evaluation aims to build evidence on how to develop collaborative reflection and action for health through multiple levels of engagement and more authentic community engagement, which has been highlighted as a key gap area for health systems research¹⁷ generally as well as in the province over the period of engagement. More broadly, it aims to enrich learning on PAR and its evaluation, as well as contributing to current debates on learning health systems¹⁸ and on health system strengthening, with many of the areas of focus of VAPAR mapping onto suggested process goals for a stronger health system.¹⁸

Methods and analysis

Evaluation approach

Participatory evaluation is a growing field,¹⁹ however, evaluation of VAPAR faces additional challenges of evaluating an intervention which is itself participatory, multilevel, multicycle, pragmatic, emergent and embedded in rapidly changing contexts. Our approach therefore includes the following elements, which reflect the programme design:

1. It is post-positivist in epistemology, recognising that knowledge is valid only relatively to a specific context, society, culture or individual and is socially (and potentially cooperatively) constructed.⁶

2. It is participatory and embedded in that it will build on reflections and insights of partners and wider stakeholders which are generated as part of the PAR cycles.

3. It will be adaptive, to allow for changes in the programme and its environment which may occur over time.

4. It is theory-based and looking for contribution (https://www.betterevaluation.org/en/plan/approach/contribution_analysis) not attribution, starting from a hypothesised theory of change and examining actual changes against that.

5. It draws from critical realist evaluation^{5 20} in trying to identify mechanisms of change operating in specific contexts, and the outcomes to which they lead.

6. The focus on actors and institutions also allows us to probe into political economy factors — incentives, power relationships, ideas and ideologies²¹ — which will be important explanatory factors for why and how change does or does not occur.

7. We will in addition record resource intensity, including intangible costs for participants, as part of good practice for thinking about replication and scalability.

It is important to note that this is an evaluation of the VAPAR approach as a whole (a meta-evaluation, rather than focussing on local actions triggered individually). Methodological points of interest and innovation will include being able to test and refine our theory of change over repeated cycles, and being able to test the adaptation of realist approaches to participatory processes.

Stages of evaluation

Developing the initial theory of change

During the first PAR cycle in 2017 to 2019, the research team developed an initial theory of change for VAPAR, based on continuous interactions with community, health system and wider public administration stakeholders over a number of prior years (in pre-pilot and pilot phases), as well as wider literature and secondary data. This theory of change considers the challenges and resources in the context, the expected causal pathways for addressing challenges, including change mechanisms and their underpinning assumptions, and desired outcomes. The expected causal pathways were discussed with stakeholders at workshops in cycle 1 and the start of cycle 2.

Testing and refining the theory

In each cycle, data will be collected which will allow us to refine our understanding of the intervention. This will include qualitative and quantitative data collected by the programme (on context, inputs, activities, outputs, outcomes and assumptions), supplemented by end-of-cycle interviews and workshops. These will be reviewed by the programme team and key stakeholders at the end of each cycle, leading to refined engagement and a more developed theory of change.

Final evaluation

This will bring together the learning from across the programme, describing the starting situation, the initial theory of change, how this was adapted over the years, the evidence of interplay of context, mechanisms, actors and outcomes, leading to the final theory of change for future testing and to inform sustainability and potential scale up and replication.

Theory of change

Figure 2 presents the initial theory of change.

Figure 2.

Initial Verbal Autopsy with Participatory Action Research theory of change. HDSS, Health and Demographic Surveillance System; SA, South Africa.

The context factors outline some of the key challenges which the programme is seeking to address — such as lack of constructive engagement between communities and health system and organisational culture issues within the health system — as well as some of the resources and opportunities, such as a progressive policy environment and a growing network of institutions collecting data on community health.

The inputs represent the envisaged contribution of the VAPAR programme in (1) supporting co-production of timely and relevant local evidence on health and other challenges faced by the communities in our focal area, and (2) enabling exchange and engagement with stakeholders within the health and public administration system to develop local solutions, collaboratively implement them and reflect on implementation.

In relation to mechanisms of change, three channels are hypothesised:

• Greater confidence in and commitment to co-producing and using evidence by all stakeholders, including as an input to services.

• Improved relationships and trust between communities, researchers and health authorities.

• Increased motivation and capacity for community involvement and localised evidence-based primary healthcare in health and other sectors.

Outputs are expected to include the establishment of a learning platform or space, ownership and uptake of locally-relevant evidence and ensuing collective action and reflection on action in an ongoing manner. A learning platform in this context is understood as a neutral, respectful forum in which to co-produce, exchange and use evidence for action, and to learn from that action.

Outcomes are broken into short-term changes, such as improved policy implementation within existing resources; medium-term changes, such as improved health service organisation, resourcing and delivery; and long-term changes, such as improved health behaviours and outcomes, but also the sustaining and transfer of the learning from the programme.

Although these are presented in a linear fashion, it is clear that these stages are connected, fluid and in continuous interaction, with the mechanisms key to bringing about change. The learning cycles present opportunities to engage in and analyse these repeated interactions over time.

Underlying assumptions were identified, many of which are themselves potentially influenced by the programme, such as:

• The research institutions, Department of Health and local communities being able and willing to engage over time and being open to dialogue.

• The three core constituencies having some flexibility of resources to be able to respond to new co-produced evidence.

• There being sufficient social coherence to support movement towards shared priorities and actions.

• There being sufficient stability in the health sector for receptivity to programme outputs.

• There being a wider interest in distributed and collaborative PAR (in relation to transfer of lessons to other settings).

Evaluation methods

The main questions to be examined by the evaluation are summarised in table 1 below, which also points to the data source for answering them.

Table 1.

Evaluation questions

Domain	Evaluation focus/indicators	Sources
Inputs
Provision/co-production of robust/timely evidence on local situation Research spaces and processes enabling engagement and exchange of local (and wider, as relevant) evidence	Degree and quality of stakeholder engagement (by whom, and for which activities; participant understanding and perception of processes) Relevance of data which is co-produced for local needs (match to known burdens of disease, link to priority and actionable topics, etc) Intensity of activities and match to programme plans (number of community meetings, meetings with health system stakeholders, etc) Use of budgeted resources by all partners Changes made to programme approach based on learning through activities	Research briefs Reflection from communities and authorities during each stage Summary programme reports End of each cycle review interviews/workshops Published VAPAR research papers Social media platforms – the conversation pieces, etc Programme budget and expenditure
Mechanisms
Greater confidence/ commitment to co-producing, using and acting on evidence by all stakeholders Improved relationships and trust between communities, researchers and health authorities Increased motivation/capacity for community involvement by health system and research stakeholders	Jointly authored outputs Regularity of meetings and other collaboration between stakeholders Changes to stakeholder perceptions of relevance of partnership and evidence Changes to stakeholder skills, engagement, confidence and self-efficacy, self-reported and as observed during interactions Changes to stakeholder relationships (eg, better communication, less hierarchical blockages and punitive relationships) Any process changes noted (including for wider Agincourt HDSS – for example, more proactive engagement with health system actors)	Participant feedback End of cycle interviews/workshops Systematic noting of observations and reflections on change from team members Important emails stored in shared drive Invitations to events between partners Stakeholder mapping by team and study participants, repeated over time to record change
Outputs
Learning platform Ownership/uptake of locally relevant evidence Collective action and learning derived from it	Continued commitment to process (eg, attendance and active participation at meetings and in joint activities – in claimed and invited spaces) Behaviour change by any of key stakeholders (greater focus on uptake by researchers, greater use of evidence by system, more community inputs into both) Collective action plans and extent of their completion	Programme reports Local action plans and follow-up reporting End of cycle interviews/workshops
Outcomes
Short-term/foundational Improved engagement researchers/communities/authorities Improved awareness of and shared local health priorities Improved healthcare processes/policy implementation with existing resources	Evidence of demand for continued exchange by all stakeholders (eg, independent meetings or collaboration, not linked to VAPAR) Value given to different forms of evidence and inclusion of different evidence in decision-making processes More evidence citation and use in local planning and review within health and other relevant sectors Inclusive strategic review and reflection used to plan and prioritise locally Changes to service planning and organisation linked to VAPAR and VAPAR-inspired processes	Programme reports and team observations Cycle 3 interviews/workshops Secondary reports, for example, district health plan, annual performance plan, integrated development plans, quarterly and annual reviews Local action plans and other PAR outputs (eg, photovoice, root causes mapping, Venn diagrams) Engagement and collaboration by other research in MRC/Wits-Agincourt
Medium-term Legitimate learning platforms to produce and exchange local knowledge Improved understanding of and commitment to equitable health priorities, including across sectors Improved health service organisation, resourcing and delivery Improved understanding of and experiences of health services by users	Perceptions of stakeholders on ownership, utility, impact on them personally of VAPAR-catalysed exchanges Any notable changes in resources mobilised for health and how these are used Reductions in reported problems for health services in study area (eg, stock-outs, unfilled posts, but also for outreach activities, for example, increased effectiveness of CHWs) User satisfaction increased, as expressed through PAR, client surveys, reduction in community protests, etc	Cycle 3 interviews/workshops Quarterly district health review reports on challenges, changes in organisation, resource use for each year Satisfaction trends (from routine reports, VAPAR data, any relevant additional Agincourt data, press reports)
Long-term Sustained legitimate learning platforms to produce and exchange local evidence Organisational culture favouring evidence of different types (within health system and research institutions) Supported decision-making to serve vulnerable and underserved populations Policy and planning informed by local evidence Improved health behaviours and outcomes Improved distribution of behaviours and outcomes Transferable process – shared learning	Continued support for VAPAR-inspired fora and activities Greater use of local data in policy and programme documents in province Clearer focus on marginalised communities in provincial health plans and reporting Engaging and relationship building across sectors, horizontally and vertically, in support of Primary Health Care (PHC) Greater access to and utilisation of essential health services Reduced morbidity and mortality Any other social impacts raised by participants Trends in inequity: health outcomes, behaviours and services will be assessed in terms of gender, age, ethnicity and income Dissemination and training materials and activities, deployed nationally and internationally Uptake of VAPAR learning and approach in other provinces of South Africa (eg, through SAPRIN, the new network of HDSS sites in South Africa or other health system research centres) and potentially beyond	PAR narratives and visual data Other relevant research reports (including VAPAR publications tracking specific health issues and training materials) Funds to sustain and progress VA, PAR, local health policy and systems research, VAPAR Provincial and district health plans and reports District health information system data disaggregated HDSS data disaggregated VA data - trends South African Population Research Infrastructure Network (SAPRIN) healthcare utilisation data Reports by programme partners (eg, WHO, StatsSA, INDEPTH network, Code4SouthAfrica)
Context
Relevant changes to context will be tracked, including: Opportunities, such as: Supportive policy and legislative environment for health service delivery and community involvement Research infrastructure exists in HDSS and expanding in South Africa Also challenges, for example, Top down/hierarchical governance in sector limits operational autonomy Low accountability to service providers and users System operates ‘in the dark’ in the absence of local data Lack of communication/trust communities and authorities Lack of power and representation of community Limited incentives for researchers to engage with health system		Policy documents, including national (eg, on NHI, PHC re-engineering and relating to relevant other sectors, such as water and alcohol and drugs) Annual provincial health expenditure data (from annual reports) End of cycle interviews/workshops Programme documents and data Wider literature VAPAR outputs, for example, tracking of decision space in the province for health) Social media and other interactions such as webinars Research infrastructure development – for example, SAPRIN/Agincourt Systematic noting of observations and reflections on change by team members News articles Other MRC/Wits Agincourt Unit research

CHWs, community health workers; HDSS, Health and Demographic Surveillance System; MRC, Medical Research Council; NHI, National Health Insurance; PAR, participatory action research; VA, verbal autopsy; VAPAR, Verbal Autopsy with Participatory Action Research.

The evaluation will be led by a team member who has some independence from the VAPAR process but who is embedded and able to facilitate reflections and learning from the main stakeholders in the research team, communities and health system. Different perspectives will be compared, noting synergies and tensions across the group. The focus will be on joint learning, and understanding the explanatory factors as much as the outputs and outcomes.

Most of the evaluation ‘indicators’ are qualitative, reflecting the focus of the programme on changing ‘software’ such as relationships, trust, attitudes and skills (for example, in communication and evidence use). Many will be extracted from routine programme and secondary data sources compiled by the research team throughout the VAPAR process.

Data analysis

Data collection and analysis will be continuous, collaborative and inclusive, with reviews at the end of each cycle to aid partner learning and inform improvements to VAPAR engagement. Results will be discussed with key actors from local to provincial level, such as community groups, PHC supervisors, facility managers, community health workers, front-line health staff, health programme and research managers and agencies in other sectors such as water and sanitation, housing and the environment.

Quantitative data such as VA will be analysed for significant trends in health indicators for different population groups. However, the bulk of the data will be qualitative and analysed thematically, focussing on understanding relevant changes, their drivers, their perceived impact on different groups and their relationships with one another, how they interact with other changing context features, any unintended consequences (including negative) and the implications for future interventions to build similar learning platforms.

Patient and public involvement

The VAPAR programme emerged from and further develops participatory action research pilots working with community members in the district, so members of the public (not patients in this context) were integrally involved in its shaping, in the selection of priority topics, in the generation of evidence and its interpretation, as well as participating in discussion and dissemination events. This approach will continue to be followed in the evaluation activities, as it is a core component of our programme and evaluation approach.

Ethics and dissemination

While we emphasise a participatory approach, in line with our programme aims, this does create a risk of an overly optimistic assessment, with both insiders and external respondents seeking to emphasise positive outcomes. This risk will be mitigated by the continuity of data collection and the multitude of sources; findings on impact on collaborators will not, for example, be drawn from a single interview but from repeated observations and interactions across time. Regular practice of reflexive and self-critical thinking will also be practiced within the research team, drawing on our own learning regarding the construction of safe spaces in which constructive, respectful reflection and critique can be encouraged.

A degree of insider/outsider tension²² is acknowledged during this process.

Risk may arise due to a lack of involvement or commitment on the part of participants in communities and the health systems, lack of data and/or controversial and negative results. Steps will be taken to promote the evaluation through regular contact and dialogue with all involved. This will be underpinned by constructive accountability to mitigate against blame, negativity and a punitive focus, embracing failures as learning opportunities and negative results as well as successes with a view to understanding key mechanisms in both.

Principles of research ethics related to un-harmful ways to treat individuals will apply. Ethical considerations are also anticipated related to the interdisciplinary, relativist and transformative nature of the work, the less rigid distinctions between researchers, implementers and advocates and the commitments to knowledge for action.

In each phase, there will be actions to minimise potential harm or negative consequences to participants. Informed consent will be gained from all participants. Participants will be informed about the nature of the research, its aims, objectives, procedures and outcomes. Participants will be assured that identifying information will be anonymised, and will not be disclosed beyond the research team without permission. Preliminary results will be fed back to, and verified with, participants before being disseminated more widely. Participants will be reimbursed for time spent participating in the research via provision of subsistence and travel expenses. All participants will be free to leave the study at any time and for any reason. Efforts to develop partnerships and processes beyond the programme will be sought throughout.

Ethical arrangements that apply to the routine surveillance in MRC/Wits Agincourt Unit will apply to the VAs that will be acquired in the proposed research. Specific ethical issues relate to the PAR and health systems elements. PAR is underpinned by relativist and transformative epistemologies, and is a dynamic and context dependent process. These features may be unfamiliar to, or viewed as unscientific by, medical research ethics committees and discussion may be necessary on these features, which may incur delays in the ethical approval process. Furthermore, PAR is concerned with transferring power through the research process towards those most directly affected by the issues investigated.^{23 24} Ethical conduct is therefore considered in terms of continual checking and rechecking of categories and dynamics of power by those intended to benefit from the process. It is also acknowledged that the changing of situations of social exclusion through empowerment gained via learning from knowledge and action may be open to stigmatisation and negative consequence in communities for individuals involved. Potential risks from participating in the long-term acting on information will be explicitly considered with participants, investigators and the International Steering Committee (which also supports quality assurance within the programme). We will work with the MRC/Wits Agincourt Unit's Public Stakeholder Engagement Office in the event of disputes or other difficulties in transparent and constructive dialogue with communities or other stakeholders to address and resolve issues where necessary.

In the PAR and health systems consultations, protecting the identities of participants may not be possible or necessary. Time will be taken at the outset with participants to ensure that ethical principles are agreed, respected, implemented, revisiting fit and function during the process. Ethical challenges also arise related to anonymity and confidentiality of visual data. Participants using visual methods receive training on how and why to secure permissions from the subjects of images. Where photographic material is collected and used, separate release permissions will be secured.

The plans centralise the social nature of knowledge creation and transfer. Acknowledging that significant knowledge emerges from the combination of disparate perspectives,²⁵ time will be invested to build relationships, trust and shared understandings where partners accommodate and learn from different cultures and systems and where control is shifted as far as possible to stakeholder groups. It is acknowledged that operating between diverse sectors and perspectives may introduce problems. Where intersectoral tensions are identified, they will be supportively and constructively addressed. The process will invest in understanding and aligning perspectives, acknowledging and accommodating differences and distinguishing roles, with the overall purpose of identifying and progressing collective agendas through partnerships that span boundaries for positive change. Potential difficulties will be mitigated against through a supportive and well-structured process, protected time, reliable data, shared dialogue, effective training and dedicated staff underpinned by principles of two-way learning.²⁵ If conflicts, tensions or problems ultimately threaten the process, ad-hoc sessions (in person wherever possible) will be held. The focus will be to respect and safeguard the partnerships. If the process fails, then reasons why and lessons learnt will be documented as a contribution to the methodological literature.²⁶

Acknowledging the relevance of where and how outputs are disseminated, reporting will be balanced between academic-practitioner literature and public media. Partner voices will be given space wherever possible, seeking to privilege the ‘local gaze’²⁷ and provide lessons which support further local action and benefits. Quantitative data on VA will be available for scrutiny through Agincourt HDSS, however qualitative data will be curated by the programme team. Findings will be shared through local meetings, briefs, social media sites, conferences and academic publications. Wider collaboration and lesson sharing with other centres in South Africa and beyond which are testing learning health system models is also planned.