Impact evaluation to communicate and improve conservation non-governmental organization performance: the case of Conservation International

Abstract

The rising prominence of more rigorous approaches to measuring conservation outcomes has included greater adoption of impact evaluation by conservation non-governmental organizations (CNGOs). Within the scientific literature, however, little consideration has been given to the unique and specific roles of CNGOs in advancing impact evaluation. We explore these issues in the context of one CNGO—Conservation International (CI)—and its experiences producing, using and funding impact evaluations over the past decade. We examine the contributions of impact evaluation to CI's mission at three different stages of CI's strategy: innovation, demonstration and amplification. Furthermore, we review incentives and barriers encountered by CI in its 10+ years' experience in impact evaluation. More coordinated and strategic use of impact evaluation by CNGOs would facilitate learning and promote accountability across the conservation community.

1. Introduction

Over the past 30 years, non-governmental organizations (NGOs) that seek to conserve biological diversity have rapidly expanded in size, abundance and influence [1]. We define conservation NGOs (CNGOs) as entities that aim to protect, manage and restore unique and threatened biodiversity, ecological processes and ecosystem services. Diverse in scope and size, CNGOs play key roles in advancing and achieving biodiversity conservation through (i) delivery of on-ground conservation actions, (ii) transfer of overseas development assistance to high-biodiversity regions in developing countries [2,3], (iii) promotion of key issues in international and national policy arenas and (iv) advocacy among the general public through outreach and education [1]. CNGOs are important innovators and early adopters in conservation science, policy and other practices, demonstrated by their development and application of new tools, methods and approaches such as spatial prioritization (e.g. ecoregional assessments [4]), debt for nature swaps [5] and ecosystem monitoring systems (e.g. Ocean Health Index, [6]).

The CNGOs are increasingly interested in monitoring and evaluation (M&E), parallel with (and influenced by) growing interest in M&E across the conservation sector [7,8]. CNGOs use a range of M&E approaches [9], including impact evaluation—the systematic process of assessing the causal effects attributable to a project, programme or policy vis à vis a counterfactual scenario [10]. The counterfactual is commonly established using a comparison group, or control, which is identical to the population or site, but is not subject to the intervention. Impact evaluations apply experimental (i.e. randomized control trials) [11], quasi-experimental (i.e. mimicking a counterfactual baseline as controls through statistical methods) [10] or non-experimental designs with no control group using a simple pre- and post-comparison [12] as well as advanced causal inference methods such as structural equation modelling and Bayesian network analysis [13,14].

Impact evaluation is recognized as a valuable tool for retrospective examination of effects of past conservation efforts and prospective assessment of future priorities and emerging strategies. Though CNGOs and the broader conservation sector have increasingly embraced impact evaluation [8,15], little consideration has been given to the unique and specific roles of CNGOs among the diverse array of conservation and evaluation actors. Rather, the scientific literature often speaks in general terms regarding the ‘conservation community’ [15] and the incentives and constraints for impact evaluations [16–18]. Considerable attention has also been given to the potential for impact evaluation to advance both scientific knowledge and conservation practice [19], but little attention has been given to the organizational arrangements and processes by which these evaluations occur. Though Mascia et al. [9] allude to specific actors involved in various forms of conservation M&E, further exploration is required to better understand roles of diverse actors in the production of salient, legitimate and credible impact evaluations [20,21]. Critical to more effective use of impact evaluation within ‘the conservation community’ is also understanding the contributions of impact evaluation to CNGO objectives, the contributions of CNGOs to the field of impact evaluation and the niche that CNGOs occupy relative to other actors.

To advance understanding of the relationships between impact evaluation and CNGOs, we explore here the current and potential future roles of impact evaluation at Conservation International (CI), an international CNGO, and the roles of CI within the field of impact evaluation. After a brief overview of CI, we describe the contribution of impact evaluation to CI's mission at each stage of CI's organizational strategy. Using examples from CI programmes and partners, we articulate the various roles that CI has taken in the development and application of impact evaluation. We then outline incentives, disincentives, barriers and catalysts encountered by CI in its application of impact evaluation. Lastly, we conclude with near-term opportunities for more strategic use of impact evaluation and highlight ways to enhance the contributions of impact evaluation in the conservation sector. The distinct types of roles, incentives and barriers articulated in this paper might assist future assessments of impact evaluation by other CNGOs and actors across the conservation sector.

2. The Conservation International experience

Founded in 1987, CI is a CNGO whose stated mission is ‘Building upon a strong foundation of science, partnership and field demonstration, CI empowers societies to responsibly and sustainably care for nature, our global biodiversity, for the well-being of humanity’. CI employs about 900 staff in its headquarters near Washington, DC, USA and 30 field offices located primarily in tropical, developing countries. In 2013, CI had an annual operating budget of $148 million, primarily funded by private foundations, public sector agencies, corporations and major gifts from individual donors [22].

To achieve global impact, CI has developed a strategy that identifies three stages to programme development and implementation: innovation, field demonstration and amplification (figure 1). Innovation involves the development and piloting of new approaches to conservation science, policy and financing, including novel tools, methods, data or evidence. Demonstration requires application of these approaches in CI's priority geographies by CI programmes and partners. Amplification involves facilitating mechanisms that promote the adoption, influence or replication of such innovative approaches by others. Global impact, the ultimate goal of CI's strategy, involves the application at global scale of innovations by CI and others based upon lessons learned in demonstration. Sufficient organizational capacity, or the technical expertise and regional knowledge necessary to ensure efficient and sustained application, underpins all three stages. Multiple CI divisions, including those responsible for science, policy, corporate engagement and on-the-ground implementation, work together to operationalize this strategy. Ultimately, organizational goals are achieved by progression from one stage to the next, though some conservation interventions might not involve all stages.

Figure 1.

Key components within Conservation International's organizational strategy to achieving global impact from its programmes and investments. Innovation involves the development and piloting of new approaches to conservation science, policy and financing, including novel tools, methods, data or evidence. Demonstration requires application of these approaches in CI's priority geographies by CI programmes and partners. Amplification involves facilitating mechanisms that promote the adoption, influence or replication of such innovative approaches by others. (Online version in colour.)

Monitoring and evaluation serves multiple purposes within CI's strategy: measuring outcomes of specific actions, managing risks, assessing and refining interventions, tracing progress towards ultimate organizational goals, supporting verification protocols, communicating and reporting results to donors and key stakeholders, and guiding future management. Ambient monitoring, for example, provides a snapshot of the national and subnational contexts within which CI operates (e.g. status of terrestrial vertebrates by Tropical Ecology Assessment & Monitoring network, [23]) and informs development of new conservation interventions (e.g. Critical Ecosystem Partnership Fund (CEPF) regional profiles, [24]). Performance measurement allows CI to track progress of its conservation interventions, as a basis for adaptive management and accountability to donors (e.g. income generation by projects supported by CI's Conservation Stewards Program, [25]). Management assessments monitor administrative capacities and activities—and may highlight the need for further investments and capacity-building activities (e.g. management outcomes in marine protected areas (MPAs) using CI's Seascapes approach, [26]). Systematic reviews capture the current state of scientific evidence for single or multiple conservation intervention(s), providing a basis for both evidence-based policy and targeted scientific research (e.g. mapping evidence on impacts of nature conservation on human well-being, [27]). Lastly, impact evaluation fosters learning and enhances accountability throughout the processes of innovation, demonstration and amplification.

Over the past 10 years, the number of impact evaluations associated with CI programmes has grown (table 1), but is still relatively small. The earliest example is a 2001 study by then CI scientist Aaron Bruner et al. [28], which evaluated the impact of 93 protected areas in reducing land pressure. While this study attempted to compare effects within protected areas and surrounding areas not under protection, the design did not account for spatial biases in the location of existing protected areas which can distort estimates of effectiveness [32]. However, the study did highlight the potential value of such investigations and was a significant catalyst, more broadly, for further analyses of protected-area effectiveness across the sector (reflected by over 1200 citations on Google Scholar as of June 2015). Concurrently, CI has invested in several non-experimental impact evaluations for accounting and learning purposes. Examples include an assessment of environmental and social impacts of coffee production certified under Starbucks' Coffee and Farmer Equity (C.A.F.E.) practices programme [33] and several outcome-based programme evaluations, e.g. independent evaluations of CI's Marine Managed Areas programme [34] and 10 years of CEPF investment [35]. The remainder of this paper focuses primarily on impact evaluation utilizing quasi-experimental and experimental designs due to the emphasis of this theme issue on identifying and measuring causal relationships between conservation interventions, and social and ecological outcomes.

Table 1.

List of impact evaluation applications associated with CI programmes and initiatives since 2001. Types of roles by CI are defined in table 2.

subject of evaluation	type of outcomes measured	timeframe	study design	scale of study	CI role	source
ecological and management impacts of 93 protected areas in 22 tropical countries	natural vegetation retained or recovered	variable. Median range = 23 years	quasi-experimental. Outcomes compared within protected areas and 10 km-wide zone outside boundaries; and between years of park establishment and date of assessment	global	producer	[28]
social impacts of 8 marine protected areas in Birds Head seascape, Papua Barat, Indonesia	economic well-being; health; political empowerment; education; culture	2010–2015+ (ongoing)	quasi-experimental. Outcomes compared with statistically matched control households	subnational	co-evaluee; co-producer	[29]
ecological impacts of 8 marine protected areas in the Bird's Head Seascape, West Papua, Indonesia	fish abundance and diversity; coral reef cover	2008–2015+ (ongoing)	quasi-experimental. Outcomes compared with statistically matched control reefs	subnational	co-evaluee: co-producer	[30]
ecological and management impacts of 65 protected-area sites, supported by Global Conservation Fund investments in 20 countries	forest cover; management capacity and activities	2008–2013	quasi-experimental. Outcomes in protected areas compared with areas within 20 km-wide zone outside boundaries using differences in differences	global	funder; producer	[3]
biophysical impacts of Indonesia's moratorium on new concessions for oil palm, timber, and logging. Comparison between place-based policies and alternative price-based instruments	reduction in emissions from deforestation	2000–2010	quasi-experimental. Outcomes compared with statistically matched control forests based upon panel data	national	producer	[31]
social and ecological impacts of protected-area network in Cambodia	land cover; ecosystem services; human well-being	2006–2012	quasi-experimental design with advanced causal inference methods. Outcomes compared with unprotected control areas	national	producer	W. Yang, M. C. McKinnon, W. R. Turner, R. Neugarten, T. M. Wright, A. Olsson, T. Farrell & D. Hole 2015, unpublished

In parallel to CI's efforts, other major CNGOs have also increasingly engaged in impact evaluation. While not including a comprehensive review here, published accounts are relatively few. Efforts to date encompass a variety of interventions including protected areas, prescribed burning, forest management, sustainable certification and payment for environmental services (PES). The Nature Conservancy (TNC) has evaluated biophysical impacts, e.g. restoration of migratory bird habitat [36], longleaf pine restoration [37] and water quality [38] in its United States programmes, and more recently, social impacts in its international programmes [39]. Other CNGOs are also evaluating social impacts of conservation. Examples are estimates of health impacts, specifically sexual risk behaviour, of community-based conservancies in Namibia [40] by the World Wildlife Fund (WWF), and PES schemes in Cambodia by the Wildlife Conservation Society [41]. CNGOs have also partnered on impact evaluations such as an ongoing study by CI, WWF and TNC on social impacts of MPAs in the Birds Head Seascape, Indonesia. Third-party impact evaluations have also been commissioned (e.g. five evaluations by Rainforest Alliance and the Sustainable Agriculture Network of their sustainable coffee and team certification schemes [42]). Both TNC and WWF have also produced technical guidance documents to assist others embarking on evaluations [29,43]. Investment in producing and commissioning impact evaluations among CNGOs is therefore growing, but is still not widespread. In its own applications, CI has placed particular emphasis on understanding mechanisms by which certain policies affect both natural and social systems and where trade-offs and synergies occur between ecosystem services and different beneficiaries [30] as well as an institution-wide process to connect these needs with a consistent metrics framework.

3. The role of impact evaluation in advancing Conservation International's strategy

At the innovation stage of programme development, impact evaluation is a useful analytical tool for exploring and experimenting with different project designs and activities. Impact evaluation can identify and articulate causal linkages within a socio-ecological system, assess the efficacy of a novel and untested model, or evaluate positive or negative effects of a programme. In these instances, impact evaluation can improve our understanding of how ecological and social systems function, or how policy, financial and management mechanisms work. This can be helpful for early screening of new and untested approaches before committing more substantial resources. To date, CI has not applied impact evaluation in testing innovative interventions or tools, but rigorous examples could be beneficial for a range of emerging initiatives including blue carbon management [44] or social impact investment models in sustainable fisheries [45].

At the demonstration stage of its programmes, CI aims to integrate up-to-date scientific findings and novel management approaches into conservation solutions on the ground. At this stage, managers want to know whether programmes can deliver intended results, and what impacts—positive or negative, intended or unintended—might occur as a result of CI actions. Donors want assurances that their investments are equitable, efficient and effective. Impact evaluation has the potential to address these needs in on-ground demonstrations. In the Bird's Head Seascape of eastern Indonesia, for example, a quasi-experimental impact evaluation of eight MPAs is providing novel insights into the ecological and social impacts of MPAs [29,30]. This monitoring effort—a collaboration among CI, WWF, TNC and UNIPA—is designed to build upon social and ecological theory, as well as best practices in impact evaluation. Best practices include BACI (before–after-control-impact) research designs and statistically representative sampling informed by power analysis to generate insights into the direction, magnitude and heterogeneity of MPA impacts between sites and populations, as well as the mechanisms by which these impacts occur. Initial results are not only advancing scientific understanding but are also informing conservation decisions at the project, programme and policy levels in West Papua including boundaries of specific MPA sites, refinement of existing management plans and integration into M&E practices and strategies of global CNGOs. Rigorous examination of impact within such in-depth case studies has the potential to establish a ‘proof of concept’ and build support for further investment and buy-in from stakeholders [9].

Demonstrations can have influence at subnational and national scales; yet alone are unlikely to be sufficient to advance broader conservation and sustainable development goals. To expand its global reach and impact, efforts by CI and many other CNGOs, research institutes and business partners seek to strengthen established operational models, draw on experiences and knowledge from site-specific case studies to other contexts, and ultimately, inspire globally influential actors to adopt and replicate effective approaches. In this context, impact evaluation has the potential to generate rigorous evidence of causal mechanisms on interventions from a few context-specific cases, provide potentially generalizable mechanisms for examination in other contexts, and ultimately help build robust guidelines and theories to guide conservation practice at broader scales [10]. In a recent 10-year retrospective, CI scientists, for example, evaluated the impacts of the Global Conservation Fund (GCF) on deforestation rates and management capacities and activities in 65 protected areas across 20 countries. Results suggest that regular sustained investment in protected-area management led to reduced deforestation rates in protected areas [3]. While the study design used by Bonham et al. had limitations (e.g. the counterfactual was within a defined distance of protected-area boundaries which has been shown to overestimate the magnitude of impacts [32]), this application has motivated support for further enhanced statistical modelling of impacts of conservation trust funds as a model for long-term financing. Impact evaluation, therefore, has the potential to support CI's strategy for amplification through provision of credible evidence on different models and mechanisms and motivation for refined evaluation methods.

Beyond measuring amplification of CI-led initiatives, CI is also interested in the extent to which efforts by the conservation community more broadly, such as protected areas or PES, are achieving national, regional and global impacts. Large-scale impacts require the collective delivery of social and ecological outcomes from interventions conducted by multiple actors, and not just CI's efforts alone. An ongoing evaluation by CI in Cambodia, for example, measures how existing protected areas affect land use and land cover, provision of ecosystem services and human well-being, as well as broader implications of proposed protected-area expansion for households living within or near protected-area boundaries. This evaluation has the potential to influence a process to reassess national protected areas and land-use plans. CI is concerned not only if a tool works, but also which tools work best when diverse options are available. In a recent evaluation of Indonesia's moratorium on timber concessions, Busch et al. [31] compared the effects of different scopes in the policy (adding secondary forests; adding existing concessions) with an alternative policy for achieving the same ends (a carbon price). CI is increasingly seeing the value of a more formative approach to evaluation, which assesses factors affecting design and delivery of a specific conservation approach to improve implementation, not only a judgement of its overall effectiveness.

4. Roles of Conservation International in impact evaluation

While impact evaluation has a critical role to play within CI's organizational mission, CI and other CNGOs also have a critical role to play within the field of impact evaluation (table 2). CNGO roles include the identification of key policy-relevant research questions [46,47], the design and implementation of impact evaluations with high potential for utilization by decision-makers [9,21], capacity-building among research partners [29] and innovation within the field of impact evaluation itself. Diffusion, adoption and adaptation (sensu [48]) have been catalysed by CNGOs ensuring that evaluation protocols, instruments and data management systems are freely available to others as open-source resources.

Table 2.

Roles of CNGOs in impact evaluation.

type	description of role
consumers	those who use or adapt methods, data, guidance or evidence generated by an impact evaluation
producers	those who implement or conduct an impact evaluation
funders	those who fund or commission an impact evaluation by a third party
evaluees	those programmes or interventions by an organization that are the subject of an impact evaluation
advocates	those who promote, facilitate or encourage the production, utilization, funding or commissioning of impact evaluation by others in their organization or sector more broadly

In addition to producing impact evaluations and innovating within the field of impact evaluation, CNGOs also serve as the subject of impact evaluations, funders and consumers of impact evaluations, and ultimately, advocates for further impact evaluation. As an implementing organization, CI projects and programmes might be the evaluees or subjects of impact evaluations. Independent researchers have evaluated pilot REDD+ projects [49] and expenditures in Africa [2] associated with CI; these studies have used non-experimental approaches without a counterfactual. CI sites have been evaluated collectively as part of global studies assessing impacts of global protected-area networks [50], though these analyses do not attribute specific impacts to CI projects or sites, or compare those impacts with interventions by other implementing agencies. The only rigorous example to date is the ongoing evaluation of MPAs in the Birds Heads Seascape discussed in §3, where CI is part of a consortium of CNGOs that support the establishment and management of MPAs.

CI also functions as a donor, commissioning and funding independent impact evaluations of its programmes and its investments. The majority of these evaluations have been commissioned by grant facilities within CI's Ecosystem Finance division. The evaluation of GCF by Bonham et al. [3] was implemented by CI's science division, so while not a wholly independent evaluation, the evaluation team sat in a separate division from staff responsible for disbursing and managing grants. Additionally, the CEPF commissioned a non-experimental evaluation (2001–2010) of cumulative effects of its grant programme, which awarded $116 million to over 1500 organizations over this time frame. The programme evaluation compared ecological and management outcomes between 2001 and 2010, but no counterfactual was used [35]. In addition to CEPF and GCF, CI has been a leader in innovative conservation finance mechanisms including establishing the Conservation Stewards Partnership, Verde Ventures and Carbon Fund. There is potential to prioritize further efforts to commission and fund impact evaluation by CI in its capacity as a donor.

CI is also a consumer of impact evaluations, using the results produced by others to: (i) guide intervention design and indicator selection by CI programmes, (ii) assess the existing evidence base for specific interventions, and (iii) adapt methods from other evaluations to CI case studies. Verde Ventures, a CI investment fund specializing in debt financing of community enterprises, has used guidance produced by impact evaluations of sustainable certification programmes (see review, [51]) to identify and screen potential projects and subsequently monitor projects against a set of customized metrics [52].

Finally, CI is also increasing an advocate for impact evaluation, encouraging and facilitating the use of impact evaluations by other actors in the sector. CI plays this role through documenting and sharing experiences and insights in publications (such as in this theme issue), conference presentations, and participation and leadership within professional networks. A growing community of practice is also involved in advocating impact evaluations and utilization of their findings including the Collaboration for Environmental Evidence (CEE), the Conservation Evidence initiative at the University of Cambridge and the Conservation Measures Partnership (CMP).

5. Incentives and barriers for impact evaluation at Conservation International

CI programmes and initiatives have encountered specific incentives, disincentives, barriers and catalysts that have inhibited and facilitated impact evaluations—each of which varies depending upon CI's role as producer, evaluee, donor, consumer and advocate. We include examples of how barriers or disincentives have been overcome or suggest potential solutions for addressing them.

As a producer and potential innovator, incentives and potential catalysts for impact evaluations by CI include willing evaluees, local evaluation partners or local expertise, viable controls and baseline data, and an evaluative question of both high scientific interest and high policy relevance [29]. Financial and political supports have been key catalysts for application of several impact evaluations. In Indonesia, CI's role as an advisor to the national government, for example, provided an opportunity to inform climate and forest policy and engage in a dialogue about potential options following from CI's evaluation of the national timber moratorium policy [31]. CI's long-term engagement in countries facilitates such policy-relevant evaluations and is likely to maximize the utilization of results by key decision-makers. New and available datasets on global trends, e.g. forest cover [53], have also catalysed recent evaluations, e.g. land-cover impacts of the Cambodia protected-area network. Sufficient expertise to manipulate these datasets and conduct econometric analyses remains a barrier, with most applications of impact evaluations primarily confined to CI's science division, though some new applications are in progress within CI field programmes, e.g. evaluation of water funds in Mexico (L. Saenz 2015, personal communication). Where possible, inclusion of in-country staff and other local partners can promote greater advocacy among key decision-makers and help realize amplification potential of evaluation results (J. Busch 2014, personal communication). In cases such as the Bird Head Seascape, strong partnerships with academic institutes and other CNGOs have helped fill knowledge gaps and technical needs.

As an evaluee, incentives and potential catalysts for rigorous impact evaluation include the opportunities to learn, to demonstrate impacts with ‘hard data’, to leverage additional funding and foster broader uptake, and to be accountable to prospective beneficiaries, government officials, donors and oneself. These benefits require strong evaluation partners and add-on financial support (rather than zero-sum funding that substitutes evaluative activities for conservation actions). Disincentives, such as the implications of exposing negative and unintended outcomes, e.g. public criticism, reduced funding support or loss of trust among key stakeholders, are significant. These must be weighed against the value of transparency and credibility. Low rewards persist for robust and honest reporting, with few donors requiring the level of rigour provided by impact evaluation. The majority of donors funding CI programmes emphasize performance measurement with a focus on reporting process (e.g. activities implemented) and outputs (e.g. reports produced, funds spent). Targeted funding support explicitly for evaluation, such as awarded by the Global Environmental Facility to its grantees, would increase the number of applications. For evaluators interested to assess CI programmes, existing barriers include inconsistent and insufficient monitoring of baseline and trend data by projects, poor data management and difficulty in accessing available datasets. Successful evaluations of CI programmes (though non-experimental in design), e.g. Marine Managed Area Program [34], were integrated into project planning from the outset and evaluators worked in close consultation with the CI project team. This approach ensured buy-in for evaluation process and utilization of results by programmes.

As a donor with potential to commission independent evaluations of its investments and grantees, incentives and potential catalysts for impact evaluation by CI include demonstrating return on investments, ensuring accountability and fostering learning among implementing entities, building scientific capacity among local partners, and learning as the basis for future conservation investments [7]. To date, commissioning of impact evaluations has primarily been focused on investment funds (e.g. CEPF, GCF). These programmes have benefitted from long-term financing spanning 5–10 years focused on specific suites of interventions (e.g. certification, community enterprises and protected areas). CI does not require its grantees to conduct impact evaluations, but all grantees must comply with standard reporting procedures which might contribute to future impact evaluations (e.g. [3]). Building on CI's experience to date, impact evaluations which target large, multi-year investments aimed at innovative conservation financing mechanisms (e.g. debt swaps, social impact investing, green bonds and pay for performance) would be a high priority. Barriers to further application of evaluation include sufficient data on ecological and social baselines and trends in control sites and populations. Evaluations are largely dependent on data derived by grantee monitoring.

In its role as a consumer of impact evaluations, incentives for CI to use outputs from evaluations by others include guidance on project design and planning based upon established archetypes and theory-of-change models, and access to standard indicators, monitoring protocols and datasets. Catalysts for adopting and integrating evaluation results into decision-making include CI's engagement in professional networks such as the CMP, CEE or ISEAL (International Social and Environmental Accreditation and Labeling) alliance for sustainability standards. Barriers for CI programmes include time required to review the literature, engage with these networks or translate generic findings into relevant site- or programme-specific guidance, though systematic reviews present one opportunity for more efficient syntheses of evidence. Processes are therefore needed that provide managers with the best available scientific information about which interventions are most appropriate [54]. Conservation practitioners have a predilection to continue doing what has always been done. Inevitably, CI programmes are likely to remain focused on nature-based interventions even if greater social impacts are possible with other types of interventions.

6. Discussion

(a). Implications for conservation science

The scientific literature has called for greater use of impact evaluations as the basis for evidence-based conservation policy [15,18] with little examination of how ‘the conservation community’ might achieve these ends. We assessed the roles played by a single CNGO, but our insights are germane to other major CNGOs. CI's experience with impact evaluation highlights five discrete, diverse and sometimes unique roles that CNGOs play within impact evaluation: producer, evaluee, funder, consumer and advocate. In each of these roles, CNGOs face specific incentives and disincentives that may (or may not) parallel those of other evaluees (e.g. corporations, government agencies), producers (e.g. university researchers), consumers (e.g. government agencies) and funders (e.g. philanthropies). Examples of disincentives include insufficient expertise or financial support to produce or fund impact evaluations on policy-relevant questions, fear of reporting failure which might have negative consequences (e.g. loss of trust or credibility among key stakeholders or donors) and lack of existing monitoring data fit for scientifically credible impact evaluation. We have illustrated how CI has tried to overcome these challenges and posed potential solutions. Further inquiry into the factors that foster or undermine salient, legitimate and credible impact evaluations for conservation actors in each of these roles would advance scientific understanding of conservation policy processes and the role of science within environmental decision-making.

Existing studies of CNGO interventions have primarily focused on the role of impact evaluation in demonstrating effects from specific case studies retrospectively (e.g. [41]). Demonstrations are an important mechanism for translating research into conservation practice and learning how different ecological and social systems function [55]. While impact evaluation of specific programmes or investments promotes accountability [7,15], by and large most needs for performance measurement could be fulfilled through other less intensive M&E approaches. Insights from CI's experiences suggest that knowledge and data generated through impact evaluation might have the greatest value for CNGOs in testing innovative conservation interventions and supporting broader amplification of effective models among influential actors. Application of prospective evaluations could be immensely valuable for predicting future scenarios, assessing risk, managing uncertainty and informing optimal resource allocation. Yet limited examples of impact evaluations used for these purposes exist to date and should be prioritized for further investigation. In general, like many other conservation practitioners, decision-making and priority setting by CNGOs is still largely based upon anecdote and experience, rather than empirical evidence [56]. Further understanding is needed on how CNGOs currently use scientific knowledge and data, how better evidence of conservation effectiveness might be used and what mechanisms might be used for diffusion of these tools, knowledge and data to other actors.

(b). Implications for conservation policy

Traditionally, calls have been made for impact evaluation to be ‘independent’ and ‘objective’ [57]. Though external or independent impact evaluations play an important role in conservation, particularly with respect to transparency and accountability, they also have important limitations. In certain contexts, CNGOs may have unique capacity to deliver impact evaluations that are salient (given their detailed understanding of decision-making environments), legitimate (given their extensive personal and organizational networks) and credible (given their rich understanding of local dynamics and conservation practices). CNGOs offer unique abilities to convene actors, channel capacities and integrate findings into policy and therefore might better serve in roles as co-producers, funders and advocates than focusing on extensive in-house production. Similarly, outputs from CNGOs may gain greater traction among decision-makers as a result of the participatory processes frequently associated with developing and implementing impact evaluations within CNGOs [29]. Thus, clear articulation of the rationale for impact evaluation (e.g. learning versus accountability) may shape decisions regarding the roles (evaluee, producer, consumer, funder) that CNGOs may play within any given impact evaluation. More generally, recognizing the diverse motivations for impact evaluation, and the diverse roles that various actors play—including the unique roles of CNGOs—has the potential to more effectively mobilize actors and to produce more salient, legitimate and credible impact evaluations within conservation.

Diverse and unique roles involved in producing, consuming and funding impact evaluation highlight the need for a collaborative network of actors, with a mix of complementary and shared capacities, to advance understanding of conservation performance and uptake of conservation insights. We have shared our own experiences, but insights from other CNGOs and other actors would contribute to better understanding of different roles for impact evaluation. For example, production of evaluations by CNGOs in partnership with academic researchers might be more efficient. While larger CNGOs such as CI have greater potential resources and opportunities to produce and fund impact evaluations, smaller CNGOs, for example, might have important roles in consuming and advocating for evaluations, or providing data to be integrated into broader evaluations. Like CNGOs, other actors will have their own unique roles and capacities beyond those we have discussed here, e.g. universities with knowledge mandates and bodies of students to develop research, or governments with legitimate authority, local constituents and policymaking responsibilities. These different perspectives need to be harnessed to raise the level of high-quality evidence across the conservation sector.

7. Conclusion

Impact evaluation is recognized as a critical means for catalysing an evidence-based revolution in conservation. As a sector, conservation lags behind other publicly funded sectors in terms of evaluation application and availability of rigorous evidence. A surge of strategic investment in impact evaluation is needed to deal not only with a myriad of new interventions at early stages of innovation and field demonstration, but also to address a backlog of widespread interventions that have largely escaped rigorous evaluation to date. Suites of interventions, including protected areas, community-based natural resource management, PES and eco-certification, are key candidates for both retrospective and prospective analyses. Impact evaluation is key to adaptive management, policy reforms, allocation of resources to sustain interventions that are working, adaptation of those not realizing their full potential and highlighting innovations whose promise is unknown. Operating at the interface of conservation science and conservation policy, CNGOs are uniquely positioned to bridge these two domains and catalyse impact evaluations that simultaneously advance scientific knowledge and inform conservation policy and practice.

Authors' contributions

All authors conceived the scope, designed the study and provided case studies and examples. M.C.M. and M.B.M. wrote the manuscript. C.B., W.Y. and W.T. provided edits and comments to the manuscript. All authors gave final approval for publication. We thank two anonymous reviewers and the theme issue editors for their inputs which improved an earlier version of the manuscript.

Competing interests

We declare we have no competing interests.

Funding

The authors are grateful for financial support from the Gordon and Betty Moore Foundation (grant nos 3519 and 3520).