“Effective social justice advocacy: a theory of change framework for assessing progress” – reflections on the terrain since its publication in 2011

This review was written after reflecting on my original article, “Effective social justice advocacy: a theory of change framework for assessing progress”, published by RHM in 2011,¹ and identifies developments in the SRHR field which would add nuance to the original paper. At its core, the original paper offers a theory-of-change framework for social justice advocacy. It describes broad outcome categories against which activists, donors and evaluators can assess progress (or lack thereof) in an ongoing manner – changes in: organisational capacity, base of support, alliances, data and analysis from a social justice perspective, problem definition and potential policy options, policy and policy implementation, visibility, public norms, as well as population-level impacts. It suggests that using these categories for evaluation enables activists and donors to learn from and rethink their strategies as the political context and/or actors change over time. The paper presents a case study comparing factors that facilitated reproductive rights* policy wins during the transition from apartheid to democracy in South Africa and factors that undermined their implementation in the post-apartheid period. It argues that after legal and policy victories had been won, failure to maintain strong organisations and continually rethink strategies contributed to the loss of government accountability and focus on implementation of new policies. By implication, evaluating effectiveness only by an actual policy change does not allow for ongoing learning to ensure appropriate strategies. It also fails to recognise that a policy win can be overturned and needs vigilant monitoring and advocacy for implementation. This means that funding and organising advocacy should seldom be undertaken as a short-term proposition. It also suggests that the building and maintenance of organisational and leadership capacity is as important as any other of the outcome categories in enabling success.

The report from which I drew this article launched me into a career shift in which I focused partly on supporting social justice funders, networks and NGOs in strengthening their strategy, evaluation and learning processes, and partly on doing external evaluations with them. On rereading this article, I asked myself “what has changed or what have I learnt in the twelve years since its publication?” In this note I focus on five areas of personal or field development, each with implications for the funding of SRHR advocacy.

Systems thinking

The paper was written with a systems frame recognising the challenge of understanding the role and contribution of advocacy by single groups when there are multiple players, all contributing in different ways and at different moments in the process to ultimate outcomes.

In the intervening years, frameworks and tools for defining the system of focus have entered into mainstream evaluation discourse and practice. Most helpful is guidance in defining the system both when shaping strategy and when evaluating, by asking: what are the key parts or actors, what are each of their perspectives, what are the relationships and interactions between these, and what are the structures that shape those interactions – meaning the social norms, relations of power, policies and standards.^†,3,4 This is helpful in defining the boundaries of the system that will be the focus of advocacy; in helping advocates clarify the forces at play in a particular context, and where they can best intervene. Identifying the boundaries of the system from the perspective of funders, advocacy groups and community groups can also be key in clarifying perspectives and building consensus for action.⁵ Using this perspective, what the paper’s case study shows is how the advocacy for policy change cast the boundaries of the system very widely, engaging individuals and institutions far beyond the health system that would have to deliver sexual and reproductive health services. But for implementation, advocacy groups tended to focus more narrowly on the public health system. Yet it was social and religious forces and norms that would stigmatise abortion providers and make health system managers reluctant to implement the law. It was those forces playing out in the family, the health system and in the education system that would fail to support young people in building their capacity to navigate their sexual relationships in ways that protected their own and their partners' dignity, rights and health. Those forces would fail to challenge the norms of transactional sex that rendered young women in particular vulnerable to HIV and unwanted pregnancy, and of sexual violence that had the same result, as well as, conversely, the damaging impacts of criminalisation of sex work. These norms, including the valorisation of motherhood even for very young women, plus a failing economy and few job prospects, contributed to young women not using contraception or seeking abortions despite these being safe and free, albeit, over time, offered at fewer public health clinics.

There is a deeper analysis needed here but for my purposes, the issue is that civil society organisations would have needed to sustain very wide-ranging alliances and do deep community-based work on social norms and values, in order to ensure that the progressive laws were actually implemented, that society at large took a stand against gender-based violence and sexual exploitation, and that young women in particular saw value in using contraception and seeking abortions when needed. Moreover, they would have had to keep attention to and develop strategies to address the well-resourced and planned anti-choice interventions from US-based initiatives that have increased their scope and influence in South Africa over time. Hence the value of mapping the systems at play as the basis for shaping advocacy strategies.

The implications for funding are tremendous. Those funders setting the system boundary at health services failed to recognise the need to build and sustain community-level movements to address social norms, values and behaviours, that is, to define system boundaries much more widely. Indeed as a general trend, funds for grassroots and local NGOs have declined, with a handful of international NGOs receiving funds for SRHR work in South Africa, without incorporating any grounded movement-building dimensions, even while funds are pouring into the country to initiate and sustain civil society groups with an anti-SRHR agenda.⁶

#ShiftThePower

Another change in this period, which relates to the need for grounded organising and movement-building, has been the #ShiftThePower movement. This was initiated in the first instance by the Global Fund for Community Foundations^‡ in 2016 to bring together multiple actors at global and local levels who were already questioning and taking action to “centre people and communities as actors, decision-makers and investors in their own development processes and societies”,⁷ as laid out in the #ShiftThePower Manifesto for Change. In particular, it has further coalesced the debate and action among international NGOs and funders on where they “fit” into the development and social justice system and how their roles should support rather than shape the agendas of those on the ground. At the same time, it has created a platform and opportunities for shared learning among a wide range of community-based organisations – from local and national foundations to youth groups to women’s rights organisations and artists’ collectives. Language to frame this approach is developing, focusing not only on local resourcing, but also on the building of agency and trust as core to the ability of communities to shape and deliver on their own agendas,⁸ and to negotiate for resources – whether with governments or funders or corporations – on their own terms. This period has also seen the proliferation of participatory grantmaking,^§ in the SRHR field modelled as early as 2009 by UHAI, the East Africa Sexual Health and Rights Initiative, East Africa's first indigenous activist fund for sex workers and sexual and gender minorities, and now burgeoning among both global and local funds that focus on or include SRHR.

One of many challenges here is that this kind of local approach can be reified as the only ethical way of supporting movements, although it too suffers from many of its funders not providing long-term general support to the participatory grant-making groups, that they can then pass on to grantees. In addition, movements require a wide diversity of capacities and relationships in addition to well organised constituents – for example research, litigation capacity. Movements are what Batliwala describes as “meshworks”⁹ to effectively influence change, and this means multiple forms of resourcing are needed.

Methodology

Participatory funds, as well as other funders resourcing small local initiatives, are leading the way in focusing on storytelling as a means of sharing and learning, and in some cases, as an alternative to funders requiring groups with few or no staff, and often with the funders’ language being their second or third language, to be writing reports to funders.¹⁰ Some are highly purposeful in shaping their approaches to explicitly change the donor-community evaluation power dynamic. Even beyond participatory funds, there is a growing interest in “trust-based philanthropy” and multiple examples of interview and conversation tools that replace pre-set reporting templates with set indicators and assumptions.¹¹ An example of an entire upending of top-down funder assessment approaches is the “pemakna” approach of the Indonesia for Humanity (IKa) fund which drew on Measuring What Matters,¹² one of the key resources of #ShiftThePower. Kamala Chandrakirana of IKa explains, “We want our process to serve as collective spaces for shared learning that engage in a relational approach based on trust and mindfulness.” Identifying evaluators who are embedded in the community, IKa invites them by asking “Do you want to join us in building knowledge from the ground up on the work of social transformation?”¹³

Outcomes-focused evaluation

One of the many challenges to funders of inviting grantees to share stories of change, is in how to use them to understand if and in what ways multi-grantee or movement funding portfolios as a whole are making progress towards achieving their objectives. Indeed, this challenge holds even when grantees are reporting against templates that ask groups to specify their outcomes, rather than through interviews or storytelling. It is therefore not surprising that there has been a huge growth in use of Outcome Harvesting (OH) as an evaluation approach – illustrated, for example, in How has work funded by Comic Relief’s Power Up programme contributed to shifts in women and girl’s power?¹⁴ which uses OH to identify in what ways women had built and influenced power across a cohort of more than 30 women’s rights groups from different countries. Conceptualised within a systems framework, OH argues that precisely because of the multiple actors, perspectives and relationships operating in shifting contexts, advocacy initiatives cannot predict their outcomes and hence it is appropriate to rather ask what actually happened (an outcome), and then trace backwards to see if an initiative’s activities plausibly influenced that outcome, whether directly or indirectly, and without any expectation that that initiative alone would have influenced the outcome.¹⁵ Indeed this and other outcomes approaches such as Process Tracing¹⁶ enable one to analyse the contributions of different players to the same outcome.¹⁷ Outcomes can be identified in multiple ways ranging from participatory storytelling processes to reviews of annual reports. Over the last five years, increased numbers of social and environmental justice-oriented funders, including SRHR, have advertised evaluation terms of reference that ask for Outcome Harvesters, contribution analysis, process tracing or similar. These methods have also received some critique because of how they quantify qualitative data, and there’s a vibrant community of practitioners working to bring conceptual clarity to some of the tensions identified. However, their focus on articulating both outcomes and how an intervention contributed to influencing each outcome with a very high level of specificity, provides data about both what happened (the outcome) and the contribution of one or a number of actors, that can be verified whether through Outcome Harvesting’s “substantiation” with external actors, or by subjecting contribution claims to tests of the level of confidence the evidence offers and/or their relative importance in comparison to alternative explanations. In the words of Lynn, Stachowiak and Coffman,¹⁸

“When done well, causal analysis can lift up and leverage the power of stories, lived experiences, and multiple ways of knowing. … When we do not use causal analysis, … we lose the ability to test our assumptions, create knowledge about effectiveness that can drive future work, and break through our cognitive and implicit biases.”

Many SRHR groups are starting to use Outcome Harvesting themselves, at times initially in order to strengthen their reporting to funders, but then finding that it helps them to unpack in what ways their strategies are or aren’t working well, whether their assumptions hold, and what strategic shifts they may need to make. For example, the African Women’s Development Fund strengthened the capacity of grantees, including those working on SRHR, to distinguish their outputs from their outcomes and in doing so to tell their stories with greater specificity, building a stronger body of evidence to support their learning.

Principles-focused evaluation

In addition to identifying if or to what extent advocacy is influencing outcomes that do or do not show progress in relation to the overall objective of an initiative, the paper emphasises values as core to social justice advocacy – both its objectives and its processes. In particular, it stresses the importance of those most affected by the issues – in current discourse, those with “lived experience” – being central to both advocacy and its evaluation. This has significant implications for evaluation as it requires, in addition to looking for outcomes, assessing to what extent the advocacy process upholds social justice principles. In this period, literature on “principles-focused evaluation”¹⁹ has provided some guidance to support advocacy groups in articulating their principles in evaluable ways, and then to evaluate if and how well they are walking their talk. Patton’s²⁰ GUIDE on “whether principles provide meaningful guidance (G) and are useful (U), inspiring (I), developmentally adaptable (D), and evaluable (E)” gives specificity to this endeavour. Related to this has been growing attention to the need for continued reflexivity of evaluators,²¹ to hold themselves accountable in their practice, with evaluation associations investing considerable energy in collectively developing and promoting adherence to guiding principles.²² Values-driven evaluation approaches are increasingly making explicit their driving principles, for example in relation to feminist evaluation.^23,24

Emergent learning – integrating strategy, evaluation and learning

A continuing challenge, however, is the separation of “M&E” from strategy, with the very words leaving activists feeling disempowered. In the minds of many advocates, and in the tone of many funding report templates, “M&E” continues to be something that someone else does onto community or advocacy groups or that such groups do only for upwards accountability to funders. One of the main arguments I put to groups that contract me for evaluation support, is that every strategic activist is continually evaluating the effectiveness of their actions, learning from that, and adapting their strategies accordingly – that strategy, evaluation and learning go together. That when they run an event, or work on a campaign, they are continually asking “how did that go? What worked, what didn’t and why? What should we do differently next time? What should we continue to do?” This is the piece of “evaluation” that frequently gets lost in the focus on indicators and fitting into predictive templates – the process of using the evidence of what happened to think and debate and learn, and from this, to continually restrategise. What has grown in this period is the articulation of this “emergent learning” process and method.²⁵ There are two key implications here. The first is that the evaluation-for-learning process cannot belong to outsiders – those who are doing the work are best positioned to look at the evidence of what happened, and work out what it means for the advocacy going forward. Even when an advocacy group or funder hires an external evaluator, the expectation that they will do the evaluation, write a report and go away, misses the entire value of evaluation, which is to provide evidence to support learning which can only be done by those involved in the work. While an external eye may see things and have insights that the group itself hadn’t seen, the learning has to be done by the advocates themselves. This is well articulated by Wenger-Trayner,²⁶ who put the experience of agency and meaningfulness at the core of social learning processes. The second, and related, dimension is that stopping to reflect and learn and review strategies ought to be built into the fabric of day-to-day advocacy processes and, in the case of organisations, into job descriptions. Time needs to go not only into gathering and documenting the evidence, but into reflecting – the “L” in the MEL acronym – monitoring, evaluating and learning.

This is seldom budgeted, especially for those advocacy groups funded by project. There’s often no money to look back and really unpack what happened, let alone to stop and reflect on its implications. In the SRHR field, many groups are expressing a growing interest in feminist evaluation which emphasises that it should be by and for participants in an initiative, and yet few of these groups can actually resource the learning process and few funders ensure that the grants they offer are realistic about the resources needed to trace changes over time, often well after a training or advocacy process is completed, and for the group(s) concerned to reflect on findings in relation to their work going forward. For an evaluator committed to fostering social justice in their processes, this often creates insurmountable conundrums as groups propose feminist principles, but cannot find the time to practise them.

Final thoughts

Looking specifically at SRHR, many of the funders focused on specific populations such as the women’s funds, sex worker and LGBTQI funds, are exploring and innovating in their approaches to evaluation and learning. They are still seeking a workable approach to building solid evidence as the basis for learning in a way that is empowering and meaningful to their constituents. But there remain key funders of reproductive rights, in particular, who are asking advocacy groups and networks to commit in advance to set actions in set periods with set indicators, which is not only burdensome but inappropriate for advocacy initiatives. Ironically, this disempowers the groups they are funding to build power, and thereby stifles the ability of grantees to respond to opportunities and needs as they arise. Given the paucity of funding for grounded organising and networking including at local, national, regional and international levels, such groups tend to still take the funds and grapple with their evaluation requirements, rather than refusing to be pushed in inappropriate directions, a trend noted by Schlangen and Coe.²⁷

One last reflection is that the publishing on evaluating advocacy, and its implications for funders, continues to be dominated by Western journals and authors. Much of the innovation underway, particularly in the global South but also by practitioners in the global North, both in SRHR advocacy and in evaluation and learning approaches, is not written up at all, let alone in the academic literature. Another role funders could play is to encourage and fund groups to document their innovations and share them with the field, both online and in journals. This would be a contribution towards the decolonisation agenda, that has gained more traction in the last decade, in particular the need to recognise multiple ways of knowing and making meaning.

Disclosure statement

No potential conflict of interest was reported by the author(s).