A fuzzy logic expert system for evaluating policy progress towards sustainability goals

Abstract

Evaluating progress towards environmental sustainability goals can be difficult due to a lack of measurable benchmarks and insufficient or uncertain data. Marine settings are particularly challenging, as stakeholders and objectives tend to be less well defined and ecosystem components have high natural variability and are difficult to observe directly. Fuzzy logic expert systems are useful analytical frameworks to evaluate such systems, and we develop such a model here to formally evaluate progress towards sustainability targets based on diverse sets of indicators. Evaluation criteria include recent (since policy enactment) and historical (from earliest known state) change, type of indicators (state, benefit, pressure, response), time span and spatial scope, and the suitability of an indicator in reflecting progress toward a specific objective. A key aspect of the framework is that all assumptions are transparent and modifiable to fit different social and ecological contexts. We test the method by evaluating progress towards four Aichi Biodiversity Targets in Canadian oceans, including quantitative progress scores, information gaps, and the sensitivity of results to model and data assumptions. For Canadian marine systems, national protection plans and biodiversity awareness show good progress, but species and ecosystem states overall do not show strong improvement. Well-defined goals are vital for successful policy implementation, as ambiguity allows for conflicting potential indicators, which in natural systems increases uncertainty in progress evaluations. Importantly, our framework can be easily adapted to assess progress towards policy goals with different themes, globally or in specific regions.

Electronic supplementary material

The online version of this article (10.1007/s13280-017-0998-3) contains supplementary material, which is available to authorized users.

Introduction

Modern sustainable development goals explicitly include minimizing cross-scale human impacts on biodiversity and ecosystem services (UN 2015), and, concurrently, ecosystem management plans increasingly integrate social and economic dimensions (IPCC 2014; Steffen et al. 2015). This more adequately considers real-world contexts but presents the challenge of developing strategies that co-achieve complex social–ecological goals (Singh et al. 2017a) that further require reliable and transparent frameworks and metrics to evaluate progress towards specific objectives (Marques et al. 2014; Nilsson et al. 2016).

Contextualizing indicators into states (e.g., mangrove coverage), pressures (e.g., rate of coastal development), benefits (e.g., yearly fish recruitment, carbon sequestration), and responses (e.g., protected area coverage, alternative livelihood investment) has been useful for tracking progress towards conservation policies (Tittensor et al. 2014), evaluating linkages between objectives, and assessing effectiveness of achieving stated goals (Nilsson et al. 2016). Mechanistic processes of impacts can also be included to link human activities and other drivers of change with stressors on species and ecosystems (i.e., the Driver Pressure State Impact Response, or DPSIR, approach; Curtin and Prellezo 2010). Tittensor et al. (2014) used such a framework to evaluate global progress towards the Convention on Biological Diversity (CBD) 2020 Aichi Biodiversity Targets (ABTs) (UNEP 2010) by averaging trends in various global indicators. The authors found that biodiversity conservation objectives are unlikely to be met by the target year, but mostly estimated general trends as opposed to quantifiable progress because of uncertainties in data and definitions of success.

There are three key challenges in evaluating sustainability policy progress. First, the lack of quantitative benchmarks for complex sustainability goals makes it difficult to evaluate how close we are to achieving policy success, particularly when goals are ambiguous, broad, or do not fully reflect stakeholder objectives (Tallis et al. 2010). Even sophisticated evaluations are thus often inherently limited to quantification of the direction and magnitude of change (e.g., Tittensor et al. 2014). Second, evaluations are challenged by uncertain or incomplete data (Joppa et al. 2016), requiring the use of proxy and composite indicators (Fulton et al. 2005). Third, quantitative ecology methods have difficulty incorporating qualitative expert local and traditional knowledge (LTK), thus excluding highly useful information and hindering collaboration with communities at the crux of social–ecological dynamics (Perera et al. 2012).

Here, we develop a fuzzy logic expert system—a program that emulates decision-making according to set decision rules—to address such methodological challenges in evaluating progress towards complex sustainability goals. Fuzzy logic has the advantage of formally dealing with the uncertainties, incompleteness, and interpretation of indicators to address a given problem (Zadeh 1965), and has been applied to evaluate ecosystem-based management plans (Adriaenssens et al. 2004), the vulnerability of species to climate change (Jones and Cheung 2017), global jellyfish abundance trends (Brotz 2011), usefulness of conservation strategies (Andriantiatsaholiniaina et al. 2004), and the environmental sustainability of human development (Phillis and Andriantiatsaholiniaina 2001). In a management context, fuzzy logic methods are also accepted by the International Union for the Conservation of Nature (IUCN) to establish criteria and inform the classification of endangered species (Akçakaya et al. 2000).

Fuzzy logic departs from traditional binary logic (e.g., 0/1, hot/cold), allowing a single value to simultaneously be a member of multiple overlapping states (e.g., “warm” and “hot”). This allows for uncertainty or differing perceptions when making inferences. For example, although temperature can be precisely measured, the perception of hotness/coldness at a certain temperature (say, 20 °C) varies between people. Fuzzy logic allows for the association of both “hot” and “warm” categories to the same temperature (fuzzification). Association to each category triggers different sets of rules that result in multiple, possibly contrasting, signals, again with different degrees of association (fuzzy inferences). Finally, these signals are combined into a single metric and conclusion (defuzzification). The design and inputs to this framework (fuzzification, fuzzy inferences, and defuzzification) are based on available data and expert knowledge (Adriaenssens et al. 2004). Expert elicitation techniques can be used to establish evaluation criteria (Mackinson 2001; Burgman 2005; Martin et al. 2012) and to provide input when no other data exist, but must be rigorously implemented to avoid biases (Perera et al. 2012).

There are multiple useful methods to monitor biodiversity or environmental state at national or subnational scales, such as the Ocean Health Index (OHI; Halpern et al. 2012), Nested Environmental Status Assessment Tool (NEAT; Uusitalo et al. 2016), and the HELCOM Biodiversity Assessment Tool (BEAT; Andersen et al. 2014), some of which are quite flexible in using multiple types of data (e.g., Backer and Leppänen 2008) and some specifically attempting to approximate the calculation of uncertainty of fuzzy logic methods (Newton 2010). The fuzzy logic system developed in this study, however, is not intended to evaluate environmental state—though it could conceivably be used to that end—but the progress status of an interdisciplinary policy goal that may include environmental state as a component of desired outcomes. Indeed, outputs from existing environmental indicator frameworks would be ideal inputs for this evaluation.

There are four key qualities that this fuzzy logic approach capitalizes on: (1) the ability to combine multidisciplinary and multiscale data, which is vital for new integrated policy goals; (2) transparency of the entire framework (assumptions, data considered, scoring mechanisms, etc.), which allows for and indeed encourages multi-stakeholder collaboration; (3) flexibility to make adaptive modifications to assumptions or data inputs, while still maintaining consistent evaluation criteria with results that can be compared over time and across regions; (4) making uncertainty and differences in perceptions explicit, which is important for integrated policies that must necessarily deal with multiple uncertainties in data, system linkages, and stakeholder preferences and objectives. To be clear, the principles above guide the development of many environmental indicator frameworks (e.g., Halpern et al. 2012; Andersen et al. 2014; Tittensor et al. 2014; Uusitalo et al. 2016); for fuzzy logic approaches, however, these principles are essential components rather than desirable traits (Akçakaya et al. 2012).

The fuzzy logic expert system developed in this study explicitly integrates different types of indicators to provide a quantitative progress score for environmental sustainability goals, in this case CBD Aichi Biodiversity Targets (UNEP 2010) specifically within marine systems. The guidance of national and intergovernmental environmental policies is important to achieve global sustainable development, but requires formal methods for transparently evaluating policy progress and identifying research gaps and investment priorities. We examine the sensitivity of this method to data variability and assumptions of model structure, and discuss potential applications of this fuzzy logic framework to other environmental and sustainability policies, such as the UN Sustainable Development Goals (UN 2015).

Materials and methods

We developed a fuzzy logic expert system to score environmental policy progress based on the direction and relative magnitude of trends for diverse biological and socio-economic indicators. The criteria and framework of this model are described in depth here, but we first provide a detailed primer on the fuzzy logic expert system key terms and evaluation framework which form the basis for our subsequent model.

Fuzzy logic expert systems

The main components of fuzzy logic expert systems include the problem (i.e., progress towards biodiversity targets), its corresponding space (i.e., potential indicator values), and criteria informing conclusions (Cox 1999; Fig. 1). Each criterion has its own space of possible indicator values and is divided into overlapping fuzzy sets (heuristic categories) that each comprise a range of values (or domain) (Fig. 1a, b). Indicator values have different degrees of membership [0,1] in corresponding sets which are determined by fuzzy membership functions. These functions are most often triangle distributions except for sets at the upper and lower bounds of the criteria or problem space, where they take on a trapezoid shape (i.e., any value beyond these limits will have a membership = 1) (Fig. 1a, b). Note that membership functions could be assigned any shape, including normal or gamma curves, or irregular user-defined functions (Wood et al. 2007) that could be more suitable depending on the application (Cox 1999).

Fig. 1.

Illustrative diagram of algorithms used in reasoning through fuzzy logic. a, b Evaluation of criteria memberships of indicator values. In b, the example x-axis value corresponds to both the ‘Large’ and ‘Very Large’ categories, each with a membership of 0.5. c Compilation of criteria information for the problem being evaluated. Memberships for each of the criteria sets intersect their corresponding problem set (see Table 1) and x-axis values provide problem scores (e.g., Criterion 1 ‘Increasing’ set in a corresponds to Problem ‘Good’ set in c; Table 1). For triangle-shaped problem sets, memberships are assigned to median set x-axis value (i.e., Criterion 1 ‘Stable’ and Criterion 2 ‘Large,’ and Problem ‘Moderate’). d Problem fuzzy region, final score, and confidence interval given α = 0.25

Defuzzification and solution score

Criteria sets and associated degrees of membership are combined formally through fuzzy reasoning and a defuzzification process to provide a solution, in this case progress towards policy goals. First, a heuristic conclusion and its associated indicator value are matched to its membership in the corresponding criteria sets, each of which has a corresponding problem set. In Fig. 1, for example, the ‘Stable’ and ‘Increasing’ sets of Criterion 1 (‘Abundance change’) correspond to the ‘Moderate’ and ‘Good’ sets of the Problem (‘Policy progress’) being evaluated (Table 1). In this example, an indicator value of 0.1 in ‘Abundance change’ has a membership of 0.65 in the ‘Stable’ fuzzy set and 0.3 in the ‘Increasing’ set, so these membership scores are transposed to the corresponding Problem sets (Table 1; Fig. 1c). In the case of set shapes with two values for one membership (i.e., two sides of the triangle), the median score for the Problem set is assigned. The resulting problem membership distribution (fuzzy region) can be displayed graphically by truncating each problem set at its maximum membership (Fig. 1d). A confidence interval for the final problem score is calculated based on fuzzy region values at a predetermined threshold (α), indicating the minimum degree of membership (or belief) acceptable to include criteria sets in the problem evaluation (Fig. 1d).

Table 1.

Illustrative example of defuzzification of results from information shown in Fig. 1

Criteria	Indicator value	Set	Membership	Problem Set	Problem Score
Abundance Change	0.1	Stable	0.65	Moderate	50
Abundance Change	0.1	Increasing	0.3	Good	65
Spatial Scope	Ocean	Large	0.50	Moderate	50
Spatial Scope	Ocean	Very Large	0.50	Good	75

As shown in Fig. 1c, a problem solution very likely includes multiple sets; in this example, ‘Policy progress’ would be Moderate–Good. A final score (S) is then calculated by defuzzifying criteria information. The most common defuzzification technique is the centroid method (Cox 1999), a weighted average incorporating all individual criteria (i) scores (s) and memberships (m) above the α value:

$$S=\sum _i(s_i·m_i·w_i)/\sum (m_i·w_i),$$ 1

where w is an optional weighting factor applied to each criterion, such that a higher value is placed on specific criteria (for example, if ‘Abundance change’ was judged to be more important for policy success than ‘Spatial scope’). Table 1 shows the scoring details for the example in Fig. 1, which, based on Eq. 1, would result in a final ‘Policy progress’ score of 59 (Fig. 1d).

Evaluation of sustainability policy progress

Our fuzzy logic expert system builds on previous frameworks that classify indicator types into states, pressures, benefits, or responses with different desired directions and magnitudes of change (Phillis and Andriantiatsaholiniaina 2001; Tittensor et al. 2014). Criteria to be informed from each indicator were selected based on their contribution to achieving sustainability goals and their flexibility in allowing for information of different types, which is vital when dealing with complex social–ecological issues (Heink and Kowarik 2010). Given a specific policy goal, appropriate indicators, and available data, policy progress (in heuristic and quantitative terms) is evaluated through the fuzzy logic model based on consistent criteria and rules (Fig. 2).

Fig. 2.

Framework for evaluation of progress towards sustainability policy goals, including identification of indicators, fuzzy logic evaluation, progress score and key factors, and feedback for adaptive management

Our criteria include the following: (i) ‘Recent’ (change since inception of policy); (ii) ‘Absolute’ (change from earliest known state); (iii) ‘Time’ (length of indicator time series); (iv) ‘Space’ (spatial scope of indicator); and v) ‘Suitability’ (of the indicator regarding the target, used as a weighting factor). A general assumption is that longer data time series collected over larger spatial areas would more accurately reflect national-level progress relative to goal benchmarks and baseline values. Note that these criteria were developed and selected in part considering the types of data commonly available at national scales, but there are many other potential criteria that could be used in similar approaches given specific policies and contexts (Adriaenssens et al. 2004) (see “Discussion” section).

Indicators with a decreasing trend would receive a positive score if classified as a pressure (e.g., decreasing fishing capacity), but a negative score if classified as a state or response (e.g., decreasing funding for habitat protection). In our framework, indicator trends are further differentiated and evaluated through the ‘Recent’ and ‘Absolute’ criteria to reflect the fact that new policies are intended to have a new and specific effect. The ‘Time’ and ‘Space’ criteria are similar to those proposed by Brotz et al. (2012) to determine changes in regional jellyfish abundances based on multiple data sources and types. These scope criteria result in a better overall score for indicators that provide information over a long period, and reflect trends over a large spatial area (Table 2). Note, however, that depending on the application this initial assumption may be modified. For example, special consideration might be given to indicators that are relatively new and/or not updated often but are comprehensive. Similarly, local policy evaluations would place higher value on indicators for local rather than large areas.

Table 2.

Evaluation criteria rationale, fuzzy sets, indicator values, and the corresponding progress sets

Criteria	Set	Indicator values	Corresponding progress set	Metric and rationale
Recent	Very good	0.2 ≤ Ratio	Very High	Ratio between most recent value and earliest value since policy implementation. This represents progress after agreement to curtail unsustainable activity. An indicator of ecosystem state (e.g., species abundance) would have a better score the greater the positive change, and opposite for an indicator of ecosystem pressure (e.g., pollutant load)
	Good	0 ≤ Ratio < 0.4	High
	Medium	− 0.2 < Ratio < 0.2	Medium
	Poor	Ratio ≤ 0	Low
Absolute	Very good	0.2 ≤ Ratio	Very High	Ratio between most recent value and earliest available value. The current state of an indicator relative to its earliest known state shows overall progress towards target success, e.g., recent recovery rate of a wild population may be very good, but abundance still far from previous known levels. See above for handling of ecosystem state versus pressure indicators
	Good	0 ≤ Ratio < 0.4	High
	Medium	− 0.2 < Ratio < 0.2	Medium
	Poor	Ratio ≤ 0	Low
Time	Very long	12 ≤ Years	Very High	Length of data time series (years). Longer time series of indicator data allow for establishment of meaningful baselines and improved analyses and policy development. Although frequent data points along a time series are highly useful, only total length (i.e., last and first year of data) is evaluated here
	Long	8 ≤ Years < 16	High
	Medium	4 < Years < 12	Medium
	Short	Years ≤ 8	Low
Space	Large	Ocean or Nation	Very High	Spatial scope of indicator. As most sustainability targets consider a national scale, indicators available for a small (e.g., city-level) spatial area can be indicative of progress, but do not carry the same weight as the ones available for much larger (e.g., ocean basin) areas. This would change depending on the spatial scope of the policy being evaluated
	Medium	Region or Province	High
	Small	Point location or Statistical Area	Medium
Suitability	–		–	Suitability weighting rank from 1 (medium) to 10 (excellent) (re-scaled in model from 0 to 1). Due to the language and broad goals of many sustainability goals, it can be difficult to select precise indicators for each target. The suitability weighting factor allows for the use of all available data while explicitly recognizing their appropriateness. Using a weighting factor (as opposed to another fuzzy criterion) facilitates using the same indicator to inform multiple targets, where the indicator can have different suitability values for each target

Finally, the ‘Suitability’ criterion is used to inform the relative appropriateness of an indicator to evaluate different policy goals. For example, a time series of the number of whale watching operators in the St. Lawrence River might be a good indicator of human economic benefits, as well as an adequate but indirect indicator of pressure on beluga populations (Blane and Jaakson 1994); in our framework, this indicator could inform both criteria, with higher weight on the former. A detailed explanation for the criteria is presented in Table 2, and the associated fuzzy sets in Fig. 3.

Fig. 3.

Fuzzy sets for sustainability policy evaluation criteria and policy progress score

Test case: CBD Aichi Biodiversity Targets

We test the fuzzy logic expert system described above using a key international policy, the Convention on Biological Diversity (CBD) Aichi Biodiversity Targets (ABTs), which include twenty Targets to be reached by 2020 (UNEP 2010). To date (2017), 196 states have signed on to the CBD as a guide for their national strategies. As the CBD was ratified in 1993, this becomes the baseline year used in the ‘Recent’ criteria (Table 2; Fig. 3). Many indicators have been proposed as reflective of overall progress and trends (Canadian Council of Resource Ministers 2014; Tittensor et al. 2014), but not all ABTs have quantitative objectives. Indeed, the current lack of a formal process for evaluating progress (Marques et al. 2014; Tittensor et al. 2014) in part prompted this study.

Drawing from an existing metadata repository of ocean-related research available for Canada (Cisneros-Montemayor et al. 2017), we compiled indicators (n = 146) and applied our method to evaluate advancement toward four ABTs. All indicators and the corresponding references are provided in Supplementary Materials (Table S1). The test Targets highlight our method’s strengths and limitations, and were selected to include varying (a) clarity of target success, (b) availability of indicators, and (c) agreement between available indicator trends:

• ABT 1 People are aware of the values of biodiversity and the steps they can take to conserve and use it sustainably [unclear goal, few indicators, conflicting trends].

• ABT 3 Incentives, including subsidies, harmful to biodiversity are eliminated, phased out, or reformed in order to minimize or avoid negative impacts, and positive incentives for the conservation and sustainable use of biodiversity are developed and applied, consistent and in harmony with the Convention and other relevant international obligations, taking into account national socio-economic conditions [clear goal, few indicators, agreeing trends].

• ABT 6 All fish and invertebrate stocks and aquatic plants are managed and harvested sustainably, legally, and applying ecosystem-based approaches, so that overfishing is avoided, recovery plans and measures are in place for all depleted species, fisheries have no significant adverse impacts on threatened species and vulnerable ecosystems, and the impacts of fisheries on stocks, species, and ecosystems are within safe ecological limits [unclear goal, many indicators, conflicting trends].

• ABT 17 Each Party has developed, adopted as a policy instrument, and has commenced implementing an effective, participatory, and updated national biodiversity strategy and action plan [unclear goal, few indicators, agreeing trends].

Sensitivity analyses

Sensitivity analyses can reveal biases in a model framework itself, as well as data gaps and uncertainties. Our sensitivity tests thus address both the number of indicators and data values and the criteria included in the evaluation. We tested the sensitivity of results to the model structure by re-calculating scores when each of the five criteria was excluded from the analysis (Phillis and Andriantiatsaholiniaina 2001). This displays effects from the inclusion or exclusion of individual criteria, showing potential biases on the final score that may signal a need for further scrutiny of a criterion’s attributes (Cheung et al. 2005).

Sensitivity to available data was tested using a jackknife approach omitting an increasingly large (0–80 %) random set of available indicators for each target. Scores were re-calculated using each random subset of indicators, and the exercise was repeated 1000 times for each target and subset proportion. This analysis can be extended to account for the weight of specific indicators on target progress by sequentially omitting each single indicator for a given target and comparing the resulting progress scores with baseline values.

Finally, we re-calculated results under a wide range of α cut values, representing the threshold membership below which information is excluded from the analysis. A low α value indicates that any information, no matter how small a membership in a given set, is included in scores and confidence bounds. The effect of increasing α values on resulting confidence bounds reflects the agreement between indicators in a given policy evaluation, highlighting the various levels of uncertainty that can be encountered when dealing with multiple information inputs (Brotz et al. 2012).

Results

The fuzzy logic expert system developed here provided progress scores for test policies; the effects of both available data and the model framework itself were subsequently tested through sensitivity analyses. The selected ABTs with highest progress score were ABT 17 (progress score, PS = 79), ABT 3 (PS = 72), ABT 1 (PS = 71), and ABT 6 (PS = 54). For ‘Recent change’ (relative to 1993), the highest positive change was for ABT 3 (recent score, RS = 87), ABT 17 (RS = 69), ABT 1 (RS = 56), and ABT 6 (RS = 33). These scores involve a transparent integration of multiple signals for each target that are summarized in fuzzy regions showing the uncertainty surrounding each score (Fig. 4). Note that the wide array of (often conflicting) indicators results in wide confidence bounds (range at α cut; Fig. 4).

Fig. 4.

Fuzzy regions and progress scores for sample Aichi Biodiversity Targets (as numbered on panels, top right) using available data for Canadian marine systems (Cisneros-Montemayor et al. 2017). Black arrows indicate the final progress score; gray arrow indicate the recent change score; dashed lines indicate the α value and resulting confidence intervals for final progress score

Overall progress scores were robust to the criteria used, with the ‘Time’ criterion contributing to higher final scores (meaning that time series tended to be relatively long). The inclusion of the ‘Absolute’ and ‘Space’ criteria resulted in slightly lower scores, meaning that in the Canadian test case trends were worse relative to historical baselines, and indicators tended to reflect relatively smaller areas (Fig. 5).

Fig. 5.

Sensitivity of policy progress score to evaluation criteria over all sample Aichi Biodiversity Targets after omitting each indicated criterion. ‘Baseline’ shows score including all criteria. Horizontal dashed line indicates the median progress score over all tests

The sensitivity of policy progress scores to the information used was tested by omitting an increasingly large subset of available indicators from the evaluation of each target. Targets with many available indicators had more robust results than targets with less indicators (Fig. 6). Note, however, that having many available indicators also increases the chances of conflicting information, leading to high uncertainty surrounding final progress scores (Fig. 4). As further discussed below, this uncertainty can also reflect the clarity and specificity of policies themselves.

Fig. 6.

Sensitivity of final progress score to the number of available indicators (n = 4 is ABT 1; n = 125 is ABT 6). For each target, an increasingly large, random subset of indicators was omitted from the analysis. Boxes show resulting scores over 1000 iterations of random subsets. Horizontal dashed line indicates the median progress score over all tests

In addition to the analysis shown in Fig. 6, we identified single indicators that have the most weight on final progress scores in terms of percentage difference from the baseline score. The more the indicators available for a target, the less weight any single indicator has on the final score (Table 3); given the same number of indicators, increased agreement between them would decrease the influence of any single indicator on the final score.

Table 3.

Percent change from baseline target progress score if indicator is included. We show a selected subset of indicators with largest positive and negative effects on each Aichi Biodiversity Target (ABT)

ABT (indicators)	Indicator	∆ Progress score (%)	∆ Recent change score (%)
1 (4)	Canadians participating in wildlife or nature-related activities	− 11	− 21
	Whale watching tour participants	+ 13	+ 27
	Recreational fishers (active)	− 9	− 34
	Ocean Wise partners	+ 8	+ 32
3 (3)	Ratio of beneficial to capacity-enhancing subsidies	− 6	− 11
6 (125)	Number of listed marine Species At Risk	+ 0.7	+ 2
	Atlantic cod biomass (NAFO Subdivision 3PS)	− 0.3	− 0.9
7 (3)	Fisheries and Oceans Canada operating budget	− 18	− 27
	Number of listed marine Species At Risk	+ 11	+ 19

Discussion

The fuzzy logic expert system developed here allows for evaluating progress towards sustainability policy goals, highlighting the key factors in progress, and prioritizing research and collection of information. Fuzzy logic is already being used in environmental management to establish impacts and threat levels to species (Akçakaya et al. 2000; Keith et al. 2013) and habitats (Wood et al. 2007), but the method proposed here is intended to provide insights into the formulation and progress of a policy itself.

The evaluation of complex policies must always consider qualitative aspects of objectives and intent, indicator selection, and contextual factors of implementation (Tallis et al. 2010), yet a final quantitative score is highly useful for conveying outcomes to stakeholders (Andriantiatsaholiniaina et al. 2004). In our test case using four Aichi Biodiversity Targets (ABTs) in Canadian oceans, the highest progress scores were for ABT 17 (‘Policy Implementation’; Progress Score, PS = 79), followed by ABT 3 (‘Eliminate Subsidies,’ PS = 72), ABT 1 (‘Biodiversity awareness,’ PS = 71), and ABT 6 (‘Sustainable Stocks,’ PS = 54) (Fig. 4). Notably, scores for recent (since the inception of the CBD in 1993) indicator change were worse for three of the four targets evaluated (Fig. 4). The uncertainty associated with complex environmental policy targets is reflected in wide confidence bounds, though ABT 17 shows potential for high agreement even using few indicators (Fig. 4). Note that the criteria developed for this test case, although informed by scientific evidence, would in practice be further revised through stakeholder and expert consultation for application to a specific policy or region. Fuzzy logic is not a substitute for high-quality information, but an avenue to facilitate its use and integration.

Coastal communities and resource users interact most closely with marine ecosystems, and Local and Traditional Knowledge (LTK) can be a major contributor to biodiversity assessments. A major benefit of fuzzy logic systems—particularly useful for management processes related to multi-stakeholder environmental issues—is their ability to work with a variety of data types (Zadeh 1983), which facilitates the collaboration between multiple stakeholders. Expert elicitation is a valuable supporting tool for such collaboration using fuzzy logic (Mackinson 2001) and is increasingly utilized in environmental science and management (Burgman 2005; Morgan 2014). There are a variety of specific elicitation techniques, some specifically for fuzzy logic systems (Gaines and Shaw 1986; Cornelissen et al. 2003), but in general these help select indicators and define membership functions (Cornelissen et al. 2003). For example, a simple approach would elicit from experts the values of the bounds and the highest degree of membership in a heuristic category. Another method could ask experts to define the qualitative categories for specific indicator values and subsequently define membership functions based on the range of responses. In all cases, elicitation techniques based on reducing overconfidence and combating cognitive biases should be followed (for reviews on this topic, see Ayyub 2001, Burgman 2005, and Morgan 2014), requiring technical and procedural knowledge of both elicitation and fuzzy logic for effective evaluation.

Quantitative scores are highly useful for benchmarking progress and achieving policy targets, though do not eliminate the need for a careful qualitative consideration of available information, its appropriateness as indicators for given policies, the intent of policy targets, and priorities and limitations for management (Dale and Beyeler 2001; Tallis et al. 2010). The integration of the DPSIR approach reflects systems-thinking aligned with the multi-objective approaches of sustainability goals and allows for considering concurrent negative and positive impacts from a single source (Atkins et al. 2011). In our test case, results highlight lagging progress on achieving sustainable marine populations compared to the other evaluated targets, overall and since the inception of the CBD (ABT 6; Fig. 4). Results also specifically point out the positive contributions of factors like whale watching and market-based incentives to achieving targets, and detrimental gaps in specific funding (Table 3). These are quantitative results from the fuzzy logic model that can prompt qualitative discussions on the suitability of particular indicators for different policies, and can be paired with more detailed analyses of existing management and information states (Ricard et al. 2012; Cisneros-Montemayor et al. 2017).

Just as important as available information is the construction of the fuzzy evaluation framework itself, including criteria and their corresponding rules (Adriaenssens et al. 2004) (Table 2). Our method extends previous fuzzy logic frameworks used to evaluate environment-related trends (Mackinson 2000; Brotz et al. 2012), together with similar efforts (not using fuzzy logic) to gauge progress towards environmental policies (Tittensor et al. 2014). A key novel aspect is the explicit inclusion of quantitative policy progress scores as result of the evaluation (Fig. 4), which allows for formal comparisons but also stresses the proper determination of criteria and membership sets, and policies themselves (Fig. 3). As noted above, expert elicitation can be highly useful for informing fuzzy logic applications (Mackinson 2001; Andriantiatsaholiniaina et al. 2004), though must be performed carefully both in framing analyses (Morgan 2014) and for establishing specific rules (Wood et al. 2007). Although very sophisticated methods are available for eliciting such information, in terms of final results there does not seem to be much added value in creating highly complex fuzzy logic frameworks, which would require more training in order to avoid misleading input stemming from misunderstood problem structures (Cornelissen et al. 2003; Martin et al. 2012; Mcbride and Burgman 2012), so simplicity of application should be a priority.

Sensitivity analyses show that policy progress scores were robust to various model structures, with the ‘Time’ criterion resulting in slightly higher progress scores for the test ABTs (Fig. 5). Note that we did not account for potential indirect relationships between criteria, e.g., a longer time series would result in a higher progress score, but may also represent a higher baseline value for ‘Absolute’ change. The potential effects of such relationships can be readily tested, but the important point is that such linkages and the criteria used must be considered and developed to fit specific policies and social–ecological contexts (Adriaenssens et al. 2004). For example, one might include criteria related to the type of information being used, who produced it or how, yearly variability, degree of support for the use of an indicator (for example, through expert consultation or literature reviews), the relationships between multiple indicators (for example, interacting environmental drivers), etc. These region- and goal-specific refinements are crucial, as most actions in global policies (e.g., UN SDGs) are intended to be taken at national or subnational scales. Therefore, the inclusion and development of new indicators and criteria, as well as the determination of what are deemed “good enough” data, will necessarily be context specific (Andriantiatsaholiniaina et al. 2004; Svarstad et al. 2008). Nevertheless, it is possible to arrive at some general indicators that are reasonable to collect, while still being indicative of much more complicated processes relevant to policy success (Pereira et al. 2013).

The results of this study reflect three key factors in formally evaluating sustainability policy progress: availability of appropriate indicator information, and the clarity and complexity of the policy goal. Unsurprisingly, a policy target with many available indicators results in a final progress score that is more robust than one with few available indicators, so long as the indicators’ trends are relatively consistent (Fig. 6). However, when a goal is ambiguous, a large number of potentially acceptable but less specific indicators may provide conflicting evidence (Dale and Beyeler 2001). In the case of ABT 1 (‘Biodiversity Awareness’), conflicting trends likely reflect uncertainty and lack of clarity in the intent of the target itself when many types of indicators could credibly be included. (For example, do more seafood labeling programs reflect awareness in the public? If people are more “aware” of the value of biodiversity but choose not to protect it, would this constitute progress?) For ABT 6 (‘Sustainable Stocks’), however, conflicting trends rather reflect complexity. There are (sometimes competing) definitions of whether a stock is “sustainable,” but all stocks within an ecosystem could never be expected to share the same abundance trends, regardless of how these are evaluated.

These inherent barriers to “success as stated” must first be addressed through modifications of policies themselves, ideally following ‘SMART’ criteria to create goals that are Specific, Measurable, Assignable, Realistic, and Time-bound (Doran 1981). This enables the use of composite indicators that reduce complexity (Heink and Kowarik 2010) and reflect system status rather than solely that of individual species or stocks (Dale and Beyeler 2001; Froese 2004; Fulton et al. 2005; Shin et al. 2010). Fuzzy logic frameworks can be used to flag such cases, promoting the adoption of clear and achievable goals and facilitating participatory modeling to identify the indicators (and weighting values) that most impact policy success.

Conclusion

Strategic goals and associated targets—such as the CBD Aichi Targets (UNEP 2010), Sustainable Development Goals (UN 2015), and the Millennium Development Goals before them—are critical to modern sustainable development and to avoid global catastrophes, but actions and progress must be formally evaluated for policies to be truly meaningful (Heink and Kowarik 2010). In this study, we address the general lack of standardized approaches to evaluate complex sustainability policies and develop a fuzzy logic expert system to assess and score progress. Fuzzy logic is a useful framework, particularly in facilitating the transparent integration of stakeholder perceptions and multidisciplinary information in evaluation criteria (Wood et al. 2007). The growing demand for interdisciplinary sustainability goals, paired with rapid global changes, requires analytical tools that are flexible, easily updatable, and can incorporate broad (often piecemeal) data and expert input (Singh et al. 2017b). A key message that is furthermore highlighted by the results of this study is that quantitative evaluation of policy progress can only be as effective as the goals themselves are clear and achievable (Turnhout et al. 2007), and the iterative and adaptive process of policy development and formal evaluation is thus key to regional and global success.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Biographies

Andrés M. Cisneros-Montemayor

is a Research Associate at the University of British Columbia and Program Manager of the Nippon Foundation Nereus Program. His research focuses on the economics of ocean resource use, including fisheries, ecotourism, and sustainable development policy.

Gerald G. Singh

is a Senior Fellow with the Nippon Foundation Nereus Program based at the University of British Columbia. He specializes in network analyses and expert elicitation methods applied to ecosystem services and natural resource use.

William W. L. Cheung

is an Associate Professor at the University of British Columbia, Co-Principal Investigator of the Ocean Canada Partnership, and Science Director of the Nippon Foundation Nereus Program. He specializes in climate change impacts on marine ecosystems and potential mitigation and adaptation measures.