Evaluating models of expert judgment to inform assessment of ecosystem viability and collapse

Abstract

Expert judgment underpins assessment of threatened ecosystems. However, experts are often narrowly defined, and variability in their judgments may be substantial. Models built from structured elicitation with large diverse expert panels can contribute to more consistent and transparent decision‐making. We conducted a structured elicitation under a broad definition of expertise to examine variation in judgments of ecosystem viability and collapse in a critically endangered ecosystem. We explored whether variation in judgments among 83 experts was related to affiliation and management expertise and assessed performance of an average model based on common ecosystem indicators. There were systematic differences among individuals, much of which were not explained by affiliation or expertise. However, of the individuals affiliated with government, those in conservation and environmental departments were more likely to determine a patch was viable than those in agriculture and rural land management. Classification errors from an average model, in which all individuals were weighted equally, were highest among government agriculture experts (27%) and lowest among government conservation experts (12%). Differences were mostly cases in which the average model predicted a patch was viable but the individual thought it was not. These differences arose primarily for areas that were grazed or cleared of mature trees. These areas are often the target of restoration, but they are also valuable for agriculture. These results highlight the potential for conflicting advice and disagreement about policies and actions for conserving and restoring threatened ecosystems. Although adoption of an average model can improve consistency of ecosystem assessment, it can fail to capture and convey diverse opinions held by experts. Structured elicitation and models of ecosystem viability play an important role in providing data‐driven evidence of where differences arise among experts to support engagement and discussion among stakeholders and decision makers and to improve the management of threatened ecosystems.

INTRODUCTION

Unprecedented intensification of human activities is pushing an increasing number of ecosystems toward collapse (Bergstrom et al., 2021). Ecosystem collapse, analogous to species extinction, occurs when an ecosystem loses its defining ecological functions, structure, and composition and transforms into another, often novel, ecosystem (Bland, Rowland, et al., 2018; Keith et al., 2015). At local to regional scales (tens to thousands of square kilometers), assessments of the probability of collapse are applied to individual patches of an ecosystem (Newton et al., 2021). It is at these scales that decisions can determine whether a patch is prioritized for protection or restoration or that interventions would be too costly and other land uses should take priority.

Identifying whether an ecosystem is viable (a high probability of sustaining defining ecosystem features) or collapsed at a particular place and time requires selecting and defining thresholds in essential ecosystem indicators (Bland, Rowland, et al., 2018; Boitani et al., 2015). Selecting indicators and identifying thresholds between a collapsed or viable state are ideally informed by data on and empirical models of the ecosystem (Bland, Watermeyer, et al., 2018; Rowland et al., 2018). However, ecosystems can be difficult to define and applicable data are often lacking, hampering consistent and objective identification of indicators of viability and their thresholds. Even in cases where objective assessment criteria are available (Bland et al., 2017), time and data constraints increase reliance on the judgment of informed and appropriately trained individuals (Batpurev et al., 2022; Bergstrom et al., 2021). As a result, expert judgments are widely used to support formal and informal assessments of ecosystem states, vulnerability to threats and risks, and thresholds of collapse (De Lange et al., 2010; Gladstone‐Gallagher et al., 2019; Murray et al., 2016). Often these judgments are made by one or 2 individuals without consultation from a range of expertise.

An individual's experience, knowledge, and values can have a large influence on their opinions and professional judgments (Burgman, 2004; Hodgson et al., 2019). Further, systematic biases can occur across organizations, stemming from their associated social and cultural norms, values, and beliefs, which can further bias individuals (St John et al., 2019). Thus, in practice, individuals using the same assessment criteria to assess ecosystem collapse may have differing opinions on what constitutes viability and disagree on whether a system is nearing collapse (Bergstrom et al., 2021; Dorrough et al., 2021). Differences in expert opinion on ecosystem viability can result in inconsistent decisions and thus variable conservation or restoration outcomes and disagreement over appropriate policies and objectives. For example, potential impacts of urban development on threatened ecosystems have been challenged in courts of law owing to differences in expert opinions on whether the ecosystem is present or on the degree of impact. In these cases, even if experts agree that the ecosystem was once present, they may disagree on whether the ecosystem is degraded, unviable, or effectively collapsed (e.g., Commercial & Industrial Property Pty Ltd v Holroyd City Council, 2013). Similarly, individual patches of an ecosystem may be converted to intensive agriculture because they are perceived by one party as unviable (based on criteria such as patch size or condition) but perceived by another as valuable for the conservation of the ecosystem and constituent species (Tulloch et al., 2016).

To overcome problems related to variation among experts’ opinions and thus reduce bias and inconsistences in decision‐making, individual judgments can be combined in an average model that can be used for prediction and as a decision support tool (Kahneman et al., 2016; Sinclair et al., 2015). When gathering judgments through structured elicitation methods, best practice is to use large, broadly defined expert panels to manage diverging opinions and reduce bias (Burgman, 2004; Martin et al., 2012; Montibeller & von Winterfeldt, 2015). Models may also be more generally accepted when based on diverse expert panels, increasing the potential for adoption among decision makers and stakeholders (Burgman et al., 2011). Despite these beneficial features, there are relatively few examples of decision support tools that engage a broad range of experts and are being used to further consistent decision‐making for threatened ecosystems (but see Sinclair et al. [2015] and Travers et al. [2023]). A key drawback to an average model is that where opinions are highly divergent, there is a risk that average judgments may satisfy few experts and may not reflect unique, yet valid, knowledge (Morgan, 2014). Understanding whether differences among experts occur, the specific circumstances in which they occur, and whether diverging judgments reflect systematic differences among groups of individuals who share similar experiences, knowledge, or values is critical for building confidence in model use. Where systematic differences arise, targeted interventions and the development of more explicit policies could assist with decision‐making processes concerning assessment of threatened ecosystems and their viability and potential for restoration.

We used a structured elicitation with a large, broadly defined group of experts to estimate parameters for an average model of ecosystem viability that could be used to support land‐use and conservation decision‐making. Our study ecosystem was a critically endangered savannah woodland in eastern Australia (Box‐Gum Grassy Woodlands [BGGW]) that extends across a very large latitudinal gradient (∼1250 km) and that has undergone substantial reductions in the extent and condition of the ecosystem owing to agricultural intensification, clearing, and the impacts of invasive species (Prober et al., 2017). The ecosystem is a priority for conservation actions at state and national levels, and restoration will be essential for its recovery (Tierney, 2022). Our contributing experts provided a wide range of experience and knowledge and play roles in ecosystem assessment, the development of knowledge of the ecosystem, communication and engagement with land managers, and the design and implementation of policies and management strategies that affect the ecosystem.

We tested whether there was evidence that individual expert judgments of ecosystem viability varied owing to their prior expertise and current occupational affiliation. An individual's expertise or affiliation may be associated with a specific range of knowledge, social and cultural norms, values, beliefs, and behaviors (e.g., St John et al., 2019) and may influence and be reasonable predictors of individual judgments (Nordhaus, 1994). We asked whether there was agreement on the relationship between ecosystem indicators and ecosystem viability, whether an equally weighted average model based on these indicators would be acceptable to most experts, and whether there was evidence of systematic differences among types of experts. Data‐driven evidence for differences in judgments among individuals and groups could help inform the potential acceptability and utility of an average model of ecosystem assessment. Statistical models of judgments that consider differences among individuals and groups could also highlight situations where opinions are likely to be consistent and those where they diverge, which could help guide strategies and policies designed to improve conservation outcomes for threatened ecosystems. Robust, consensus‐based models of ecosystem viability are imperative for consistent ecosystem assessment and to guide the appropriate prioritization of ecosystem patches for protection and restoration.

METHODS

The ecosystem

The BGGW is a savannah woodland ecosystem with an overstory dominated or codominated by one or more of Eucalyptus albens (white box), Eucalyptus melliodora (yellow box), Eucalyptus blakelyi (Blakely's red gum), or Eucalyptus moluccana (coastal grey box), although these may be absent owing to prior clearing (NSW Threatened Species Scientific Committee, 2020). The BGGW was widely distributed on relatively fertile flat to undulating terrain throughout southeastern Australia prior to agricultural expansion in the late 19th century (Prober & Thiele, 1995). The distribution of BGGW coincides with the wheat–sheep belt, an intensive agricultural region that has been subject to clearing, pasture sowing, and cropping (Bedward et al., 2007). In addition to complete loss, the ecological composition and structure of BGGW have been altered via impacts of livestock grazing, fertilizer applications, invasion by non‐native plant species, changes in fire regimes, losses and changes in fauna composition, and recruitment failure of dominant tree species (Prober et al., 2017). The ecosystem is listed as critically endangered under Australian Commonwealth legislation (Department of Environment & Heritage, 2006) and in New South Wales (NSW Threatened Species Scientific Committee, 2020) and is thought to occupy <5% of its original range (Department of Environment & Heritage, 2006). In New South Wales, the defining features of the ecosystem are formally described in a final determination (NSW Threatened Species Scientific Committee, 2020).

Expert selection

We sought to identify and recruit a diverse range of experts familiar with BGGW in southeastern Australia. We identified 191 individuals from New South Wales, Victoria, and Queensland with potential expertise in the identification, management, or restoration of BGGW. Individuals were identified based on publishing history or nomination by state environment agency staff or other experts (Travers et al., 2023). A preliminary online survey was emailed to all individuals that invited them to participate, self‐rate their expertise, and nominate additional experts. One hundred eleven experts completed the preliminary survey, and 83 signed a consent form and agreed to participate in the online experimental survey. The final pool of experts included individuals from 4 Australian states and territories and a broad range of organizations (Table 1). Experts participating in the online survey were sent links to the final determination, explanatory information, and the survey on 21 September 2020 and given until the 8 October 2020 to complete the survey, after which date the survey was closed. Consent forms and anonymous data collection and storage methods were reviewed and approved by the New South Wales Department of Planning and Environment.

TABLE 1.

Affiliation of the sample of 176 experts initially contacted, who subsequently completed the preliminary survey of experience and expertise (n = 111) and the online elicitation of judgments of ecosystem viability.

	Affiliation	Number contacted	Responses to preliminary survey	Responses to online elicitation
	Research institution	39	17	12
	Private consultant	52	34	25
	Government agriculture	22	18	16
	Government conservation	58	39	27
	Nongovernmental organization	5	3	3
Total		176	111	83

Preliminary survey

The preliminary survey elicited information on each expert's current employment and asked them to rate the extent to which they agreed with 17 statements of experience and expertise on a 5‐point Likert scale (Appendix S1). Expertise items covered a range of land management expertise, general and specific ecological knowledge, research, and conservation policy and management. Expertise items were presented in a randomized order to each respondent.

Definition of ecosystem viability

Experts were told that a patch of BGGW is viable if the species and ecosystem processes that sustain the ecosystem, as inferred from the final determination for the ecosystem in New South Wales, are present and will persist for at least a further 20 years given reasonable management effort. Reasonable management effort was defined as standard management strategies and tactics available to land managers that are expected to maintain and improve vegetation integrity. These strategies included management of total grazing pressure, including livestock exclusion if required; annual weed control; pest animal control; retention of fallen timber and dead trees; and prevention of future nutrient enrichment and fertilizer application. Reasonable management effort excluded rehabilitation and revegetation (planting and seeding).

The above definition of viability is consistent in intent with International Union for Conservation of Nature (IUCN) conception of collapse (Bland, Rowland, et al., 2018)—that is, ecosystem changes are persistent, rather than transient, and reversal or recovery is unlikely to occur within decadal time scales in the absence of significant intervention (e.g., ecological restoration).

Elicitation experiment site selection

We selected 128 full floristic sites with vegetation that was considered consistent with that of BGGW or one of its degradation states (see below); used standard sampling methods to obtain vegetation data (visual cover estimates, 20 × 20‐m floristic site nested within a 20 × 50‐m structure site); and selected sites for which field photographs were available. Sixteen sites were excluded because field photographs and site data were inconsistent.

Each of the remaining 112 sites were allocated to 1 of 6 broad state groups: reference woodland, grazed woodland, derived native grassland, degraded woodland, degraded grassland, and thickened woodland (i.e., areas with high densities of trees or shrubs and dense overstory) (for methods, see Appendices S2–S4). These allocations and ecosystem states ensured we sampled a range of conditions and histories but were not available to the experts during the elicitation.

Site sheets were prepared that included information on the vegetation composition and foliage cover (individual species and their visual estimates of foliage cover in a 20 × 20‐m site), vegetation structure (number of trees with hollows, number of trees with a >50‐cm diameter at breast height, length of woody debris with a >10‐cm diameter, number of tree species regenerating), bioregion in which the site was sampled, sampling month, prior 12‐month rainfall, and distance from the site boundary to the nearest E. albens (white box), E. melliodora (yellow box), or E. blakelyi (Blakely's red gum). This was complemented with a site photo and a satellite image showing the location of the site in the surrounding landscape (Appendix S5). To assist experts with limited botanical skills, each native plant species was annotated with their primary growth form (e.g., grass, forb, shrub; see Oliver et al., 2019) and whether they were listed as part of the assemblage in the NSW description of BGGW.

Through the software Qualtrics (https://www.qualtrics.com), experts were each presented with 16 sites, of which 4 were common. The 4 common sites were a reference woodland, derived native grassland, grazed woodland, and a degraded woodland. The common sites were presented initially but in a random order, and the remaining 12 were drawn at random from the pool of 108 remaining sites, ensuring that experts received examples of at least 4 of the 6 state groups.

Experts were told that for each site, the data were representative of a patch of vegetation that occurs in an environmental domain with high probability of supporting BGGW. Experts were asked to provide their judgment of the probability (on a scale of 0%–100%) that the patch was viable BGGW. Experts were asked to estimate 3‐point probabilities (high, low, best guess [Burgman, 2016]) and offer a binary response (yes or no) that a minimum assemblage of sustaining species and supporting ecosystem processes (as indicated by the information provided) were present and would persist for >20 years. Experts were also asked whether the site was of conservation value or could be restored. Because judgments on all 3 of these queries were highly correlated, we present only results for viability.

Prior to undertaking their assessments, experts were required to agree that they had read both the final determination for BGGW in New South Wales and the provided definition of ecosystem viability.

Data analyses

Experts were assigned to 1 of 5 current affiliation groups: government conservation (employed by local, state, or federal nature conservation or environment departments); government agriculture (employed in government primary production, agriculture, or rural land management departments); private ecological and botanical consultancy; nongovernmental organization; and research institution (employed by universities and the Commonwealth Scientific and Industrial Research Organisation). Only 3 experts employed in nongovernmental organizations completed the final elicitation, and these were excluded from model fitting.

We derived an optimal number of clusters to represent differences in self‐assessed expertise across the 17 Likert response statements with latent class modeling with the VarSelLCM package in R (Marbac & Sedki, 2018). Three clusters were identified. The strongest discriminative power was associated with 3 statements related to expertise in land management (erosion and soil management, weed control, and revegetation) (Appendix S6). Experts in cluster 1 disagreed that they had expertise in land management, those from cluster 3 strongly agreed that they did, and experts in cluster 2 were intermediate in their response (Appendix S7). The 3 clusters were treated as a single categorical variable, management expertise, and levels corresponded to a low, moderate, or high degree of expertise in land management. All levels of management expertise were represented in each of the affiliation groups (Appendix S8).

We estimated parameters for a Bayesian hierarchical model for predicting the probability that a patch of BGGW was viable based on 1274 observations from 80 experts among 111 sites (a single site with missing site data, comprising 6 observations, was excluded from the model). The model combined 4 ecosystem indicators, expert affiliation, and management expertise and accounted for variation among all experts and sites. Numerous ecosystem indicators could be used by experts to inform their judgments of ecosystem viability based on the site descriptions provided. To minimize model complexity, we used 4 uncorrelated indicators (correlation coefficient <0.65) that were likely to be useful predictors based on evidence in the scientific literature (see below), the final determination for BGGW, and management guidelines for BGGW and related ecosystems (Good et al., 2021; Rawlings et al., 2010). The 4 ecosystem indicators were large trees (the presence or absence of trees with a >50‐cm diameter at breast height); ground layer richness (the number of native ground layer plant species, excluding subshrubs); non‐native cover (the summed visual cover of all non‐native plant species); and local tree cover (the cover of native trees within a 500‐m radius of the center of each site, derived from the satellite imagery).

We expected large tree presence to indicate viable sites because they are keystone features in grassy woodland owing to their role in habitat provision (hollows, fallen timber), nutrient cycling, and resource provision and distribution (Lindenmayer & Laurance, 2017). Large trees are a limiting resource, and in many cases, their rates of decline are greater than rates of replacement; once lost replacement may take more than 100 years (Gibbons et al., 2008).

The richness of the native plant ground layer (sum of forbs, ferns, grasses, and grass like and other growth forms) in grassy woodlands is an indicator of vegetation quality or condition, soil nutrient status, and prior land‐use intensity (Sinclair et al., 2015). We expected judgments of viability would be positively correlated with the richness of native ground layer species. Experts were not directly provided with an estimate of ground layer richness, but they had access to the list of all plant species and their growth forms.

Invasive non‐native plant species can affect vegetation structure, species composition, and ecosystem function (e.g., nutrient cycling) (Vila et al., 2011) and is a useful indicator of the extent to which a grassy woodland has been affected by agricultural land‐use practices. Non‐native perennial grasses that are unpalatable to livestock (e.g., Coolatai grass [Hyparrhenia hirta]) often invade and dominate ground layer vegetation of BGGW (Godfree et al., 2017). We therefore expected increasing non‐native plant cover to be negatively correlated with expert judgments of ecosystem viability.

The cover of native trees in a 500‐m radius surrounding a site was used to estimate local landscape native vegetation cover. Increases in landscape vegetation cover are correlated with habitat connectivity, local patch size, land‐use intensity, species persistence, and ecological functions in local habitat patches (Bennett et al., 2006; Fahrig, 2013). Increasing local tree cover was therefore expected to be positively correlated with judgments of viability. Experts had access to a standardized satellite image centered on the site that showed tree cover in the landscape.

We developed models with both the binary and continuous best estimates of likelihood. Because both data sets generated similar models and results, we present only results for the binary (binomial) response modeled with a logit link. We chose the binary data because they avoid potential ambiguity among individuals in their interpretation of a continuous probability scale and because decisions in conservation regulation, planning, and management are most often binary (e.g., yes, a site is viable and should be conserved or no, a site has collapsed and will require ecosystem reconstruction).

Our full model included normally distributed varying intercepts for each expert and site and main effects of management expertise (low, moderate, high), affiliation (government conservation, government agriculture, consultant, research), and the 4 indicators of viability (large trees, ground layer richness, non‐native cover, and local tree cover). We initially included interactions between each indicator and both management expertise and affiliation; however, preliminary fits suggested that only an interaction between non‐native plant cover and affiliation and management expertise was supported by the data.

The model was expressed as

$$Y_{ij}\sim \mathrm{Bernoulli}\left(P_{ij}\right),$$ (1)

where Y_ij is the binary response variable for site i and expert j (Y_ij  = 1 when the site is viable, and Y_ij  = 0 when it is unviable) and P_ij is the probability of site viability for expert j at site i. For the full model, the logit link function used to model P_ij was a linear combination of the predictor variables:

$$\begin{array}{ccc}& & \mathrm{logit}\left({\mathrm{P}}_{ij}\right)=\alpha +{\alpha }_{\mathrm{expert}j}+{\alpha }_{\mathrm{site}i}+{\beta }_1{X}_{1,ij}+{\beta }_2{X}_{2,ij}\hfill \\ & & +{\beta }_3{X}_{3,ij}+{\beta }_4{X}_{4,ij}+{\beta }_5\mathrm{expertise},\mathrm{moderat}{\mathrm{e}}_j\hfill \\ & & +{\beta }_6\mathrm{expertise},\mathrm{hig}{\mathrm{h}}_j+{\beta }_7\mathrm{affiliation},\mathrm{consultan}{\mathrm{t}}_j\hfill \\ & & +{\beta }_8\mathrm{affiliation},\mathrm{researc}{\mathrm{h}}_j+{\beta }_9\mathrm{affiliation},\mathrm{agricultur}{\mathrm{e}}_j\hfill \\ & & +{\beta }_{10}\mathrm{non}-\mathrm{nativ}{\mathrm{e}}_{\mathrm{expertise},\mathrm{moderate}j}\hfill \\ & & +{\beta }_{11}\mathrm{non}-\mathrm{nativ}{\mathrm{e}}_{\mathrm{expertise},\mathrm{high}j}+{\beta }_{12}\mathrm{non}\hfill \\ & & -\mathrm{nativ}{\mathrm{e}}_{\mathrm{affiliation},\mathrm{consultant}j}+{\beta }_{13}\mathrm{non}\hfill \\ & & -\mathrm{nativ}{\mathrm{e}}_{\mathrm{affiliation},\mathrm{research}j},\hfill \end{array}$$ (2)

where α is the overall intercept; β₁, β₂, β₃, and β₄ are the coefficients for the indicators of viability (X _{1 ,ij} , presence or absence of large trees; X _{2 ,ij} , ground layer richness; X _{3, ij} , non‐native cover; X _{4 ,ij} , local tree cover); β₅ and β₆ are coefficients for management expertise (low is the reference level); β₇ through β₉ are coefficients for affiliation (government conservation is the reference level); and β₁₀ through β₁₄ are coefficients for the interaction between non‐native cover and each level of expertise and affiliation. The intercepts α_{expert  j} and α_{site  i} vary for each expert j and site i, respectively:

$${\alpha }_{\mathrm{expert}j}\sim \mathrm{normal}\left(0,{\sigma }_{\mathrm{expert}}\right)$$ (3)

and

$${\alpha }_{\mathrm{site}i}\sim \mathrm{normal}\left(0,{\sigma }_{\mathrm{expert}}\right).$$ (4)

Priors for all β and the global intercept were assumed to be normally distributed and to have a standard deviation of 2.5. Default priors were used for the covariance of the varying intercepts with a shape parameter of 2.5. Before model fitting, all continuous covariates were converted to z scores (subtracting the mean and dividing by the standard deviation).

Model posterior coefficients, posterior linear predictions, and 90% credible intervals were derived from the full model to assess evidence of differences between groups of experts and to assess the effects of each indicator on judgments of viability. Convergence of all models was assessed using visual checks of trace plots and confirmation that estimates of $\widehat{R}$ were <1.05 and effective posterior sample sizes were >1000 for each parameter. All models were fit in the rstanarm package in R (Goodrich et al., 2018).

To have predictive value in new sites, expert and site intercepts and fixed effects for affiliation and management expertise must be excluded (average model). To evaluate whether this average model provided unbiased predictions, we examined in‐sample model classification metrics for each affiliation and management expertise group. Classification rates (accuracy), sensitivity (true positive rate), and specificity (true negative rate) were estimated for each group with the full model and the average model based on mean predictions from 1000 posterior samples and a prediction threshold of 0.5. For both models, posterior probabilities were estimated excluding varying intercepts for site.

We plotted and summarized in‐sample posterior predictions for all sites from the average model and a model that excluded individual and site intercepts but retained affiliation and management expertise. The later provided average predictions for each expert group and were derived to separately evaluate the effects of affiliation (averaged over all levels of management expertise) and management expertise (averaged over all levels of affiliation). We identified cases where the average model and predictions from each expert group were consistent or inconsistent.

RESULTS

Full model

Approximately 62% of all judgments (n = 1274) were positive (i.e., viable). The full model provided a good fit to the observed data. The overall mean in‐sample classification rate was 92%, had a slightly higher false positive rate (13%) than false negative rate (5.5%), and slightly overpredicted the proportion of occurrences that were viable.

Experts employed in government conservation and as consultants were more likely to judge an example of BGGW as viable, relative to the average expert from government agriculture (99% probability of a negative effect of agriculture relative to that of conservation) (Figure 1). There was an ∼85% probability that on average individuals from research were less likely to judge a site as viable (i.e., a negative response coefficient) relative to those from government conservation. When holding all other covariates at their average and in the absence of large trees, the median probability of judging a site as viable was 25% for a government agriculture expert, 44% for a researcher, 65% for government conservation employee, and 68% for a consultant (Figure 2). Model coefficients showed that differing levels of expertise had little effect on whether experts judged a site as viable (Figure 1).

FIGURE 1.

Posterior estimates for the main ecological indicators, coefficients associated with expert affiliation and levels of management expertise, their interaction with non‐native vegetation cover, and variance associated with individual sites and experts (points, means; bars, interquartile range; whiskers, 90% credible intervals). The intercept applies to an individual from government conservation with low levels of management expertise and in the absence of large trees.

FIGURE 2.

Posterior predictions of the mean probability (and 90% credible interval) that a site with and without large trees will be judged viable by an expert with moderate land management expertise from 4 affiliation groups (all other covariates held at their mean).

Judgments that a BGGW site was viable were positively correlated with presence of large trees, richness of native plants in the ground layer, and cover of native vegetation in the local landscape and negatively correlated with cover of non‐native vegetation (Figures 1 & 3). Under many circumstances, sites with large trees were approximately twice as likely to be judged viable (Figure 3). On average, sites with large trees in landscapes with >50% local tree cover, more than 20 ground layer plant species, and <30% non‐native plant cover were highly likely to be judged viable (>90% median probability regardless of affiliation). In contrast, sites lacking large trees, in landscapes with 50% non‐native cover and <30% local tree cover, and with fewer than 10 ground layer native plant species were highly likely to be judged as collapsed (<10% median probability of viability, regardless of affiliation).

FIGURE 3.

Posterior predictions and 90% credible intervals of the probability of ecosystem viability with and without large trees and given variation in (a) local tree cover, (b) native ground layer plant richness, and (c) non‐native plant cover (all other covariates at their mean). In panels (a) and (b), predictions are for an average expert with moderate management expertise from a government conservation agency. In panel (c), predictions are for an average expert with a low level of management expertise from a government conservation agency (left) and an expert with a high level of management expertise from a government agriculture agency (right).

The relationship between non‐native vegetation and ecosystem viability varied depending on expert affiliation and land management expertise. On average, individuals from government conservation and those with low levels of management expertise tended to perceive non‐native ground cover more negatively than other expert groups (Figures 1 & 3c).

Variation among individual experts (sigma, expert) was large, despite inclusion of affiliation and management expertise in the model (Figure 1; Appendices S9 & S10). Excluding affiliation and management expertise from the full model reduced the median Bayesian R ² by 0.02 (Appendix S11). Excluding both expert groups and individual intercepts reduced the Bayesian R ² by 0.16, suggesting that knowledge of an experts’ identity could substantially improve the ability to predict viability (Appendix S11). Within expert groups, there was also considerable variation in judgments of likelihood. Although individuals from government agriculture were on average more pessimistic, not all individuals were. Likewise, although government conservation and private consultants were generally more likely to consider a patch viable, some made judgments consistent with those in government agriculture (Appendices S9 & S10).

Fit of average model with judgments

The average model in‐sample classification accuracy was reasonable (82% classification rate) but less precise than the full model (88% when varying site intercepts were excluded). The average model had higher sensitivity (0.90) than specificity (0.69), suggesting it was more likely to accurately predict when the ecosystem was thought to be viable than when it was collapsed. In‐sample accuracy for the average model differed among groups of experts according to their affiliation and was highest among those from government conservation (88%) and lowest among government agriculture (73%) (Appendix S12). This difference was primarily owing to an increase in false positives and decline in model specificity among the government agriculture group (0.55 average model vs. 0.83 full model). Declines in specificity were also observed among those with a research affiliation. Classification accuracy based on the average model was similar among those experts with low and moderate management expertise (85% and 87%, respectively) but lower when expertise was high (76%), again owing to greater number of false positives and lower specificity.

Average model predictions were generally consistent with predictions from each management expertise group (Appendix S12) but less so with affiliation (Figure 4). Although the distribution of predictions was highly consistent between the average model and those employed in government conservation, they were least consistent with predictions for those employed in government agriculture (Figure 4). The latter group more often predicted a site was not viable (with prediction threshold >0.5) than the average model did. When predictions were classified as viable with a prediction threshold >0.5, differences between government agriculture employees and the average model were primarily among grazed woodlands (66% of sites viable vs. 92% for the average model, n = 38), derived native grasslands (37% vs. 68%, n = 19), and degraded woodlands (6% vs. 44%, n = 18).

FIGURE 4.

Posterior mean predictions of ecosystem viability for 111 independent sites (labeled by a priori ecosystem state groups) based on the average model and for each of the 4 expert affiliation groups based on the full model excluding random effects for site and individual (blue, predictions with a higher likelihood of viability; red, predictions with low likelihood of viability; yellow, sites with uncertain likelihood of viability [i.e., approaching 0.5]; sites sorted by posterior predictions from the average model; DG, degraded grassland; DW, degraded woodland; DNG, derived native grassland; GW, grazed woodland; TKD, thickened woodland; REF, reference woodland). Mean predictions are derived from 4000 posterior simulations of the models fitted using 1274 observations across 80 individuals.

DISCUSSION

We found systematic differences among experts that depended to some degree on an expert's affiliation and less so on their management expertise. Although affiliation did not explain substantial variation among experts, individuals from government conservation and consultants were more likely to determine that a patch of the ecosystem was viable than individuals from government agriculture. We further found that an average model tended to predict patches as viable in more instances than a model that included affiliation, expertise, and variation among individuals. These findings suggest that uncritical application of an average model may not be supported by all stakeholders involved in the development and implementation of strategies to conserve this critically endangered ecosystem. They also highlight the potential for conflicting advice about the necessary and appropriate actions for specific patches of this widespread ecosystem. More broadly, there is also potential for disagreement on design and implementation of policies and regulations intended to protect, manage, and restore it.

Despite availability of an agreed‐on and legislated definition of the ecosystem and criteria to support assessments of viability, there was substantial variation among individuals. Certain individuals were consistently biased such that they were predictably more or less likely to judge a patch of the ecosystem as viable, regardless of the specific site characteristics. Regan et al. (2005) found similar discrepancies among experts judging species extinction risk, despite providing experts with the same data and use of structured and transparent protocols. In practical terms, our results support the conclusion that reliance on a few experts, irrespective of their affiliation or expertise, could generate variable and biased judgments. Further, if different experts are relied on in different cases, advice may be inconsistent (Travers et al., 2023). These results reinforce the widespread recommendation to engage diverse expert panels and aggregate their judgments in models to improve decision‐making (Clemen, 1989; Kahneman et al., 2021).

An average model of ecosystem viability

Development of an average model of ecosystem viability holds considerable appeal for field assessment and decision‐making. Average models offer the potential for consistent, transparent, and repeatable predictions in a wide diversity of cases, thus reducing noise and increasing the reliability of decisions across similar scenarios (Kahneman et al., 2021; Sinclair et al., 2015). The average model of ecosystem viability, derived from equally weighted opinions of all experts and 4 readily sampled a priori ecosystem indicators, had an overall classification rate of 82%. This relatively simple model could provide a generalisable prediction of the viability of BGGW, despite the complexity of the problem at hand. On average, a viable patch of BGGW was associated with the presence of large trees, few non‐native plant species, diverse native ground layer species, and relatively high cover of trees in the surrounding landscape, consistent with our a priori predictions. Furthermore, except for the cover of non‐native species, the effect of each indicator was consistent among expert groups. These indicators may be collectively useful for predicting viability and could provide a basis for broader communication of risks associated with different ecosystem states. In practice, land managers, consultants, scientists, and government field officers often make in‐field judgments or provide advice about whether individual patches of ecosystems are effectively collapsed or viable. For example, in New South Wales, regional field officers must assess whether vulnerable and endangered ecological communities are functioning and viable (New South Wales Government, 2018). Although these assessments have implications for agricultural activities, decisions are often made by individuals in the absence of specific decision tools or training (Audit Office of New South Wales, 2019). Our results emphasize that these individual decisions will be highly variable. Adoption of a model or decision support tool derived from expert judgment, such as that developed here, could inform more reliable and transparent land‐use decisions (Kahneman et al., 2021; Travers et al., 2023). If extended to a spatial model, there is also capacity to support strategic land‐use planning decisions, including priorities for conservation, management, and restoration (Tierney, 2022).

Challenges for an average model of ecosystem viability

Despite the potential benefits, an average model may be less acceptable to some experts. Our average model often predicted a patch was viable, whereas our more nuanced model that considered expert, expert affiliation, and expertise predicted the patch had collapsed. Even when based on the democratic processes implicit in equally weighted and structured elicitation methods, average models could disenfranchise individuals who disagree and can obscure and delegitimize underlying disagreement and diverging opinion (Peterson et al., 2013). In practical contexts, that individuals might disagree with the predictions from an average model is not surprising or necessarily problematic unless the model exhibits bias (Morgan, 2014). Our findings, however, suggest that some groups may be more supportive of the predictions from the average model of BGGW viability (government conservation employees and consultants) than others (government agriculture employees and to a lesser extent researchers).

Our expert selection strategy was intended to be broad and included experts with agricultural and land management expertise. Despite that, our expert panel was dominated by government conservation and consultant ecologists, and the average model better reflected their judgments. Because of the potential role of expert selection on model bias, decisions about what constitutes an expert and strategies employed to select them should be carefully considered (Travers et al., 2023). Many formal assessments of ecosystem collapse rely on the judgments of a narrowly defined set of expertise, often restricted to trained scientists (e.g., Bland, Watermeyer, et al., 2018), although a much broader group of individuals could often provide useful knowledge of an ecosystem (Hemming et al., 2018). Ecologists and conservation biologists are likely to dominate expert panels tasked with assessing ecosystem collapse and potential for recovery (as in our case). If values shared by ecologists and conservation biologists differ from those of the wider population and these values influence their judgments, then disagreements may be inevitable (Velland, 2019). The exclusion of stakeholder groups from the elicitation process, particularly those who may be affected by decisions that arise from expert judgments, may also lead to questions around the legitimacy of average models, even if the predictions are reasonable (Burgman et al., 2011). For example, Indigenous knowledge holders and private land managers were not identified in this elicitation, but they are valid sources of expert knowledge of this ecosystem. Their opinion may not only differ but may provide unique perspectives, important for identifying equitable land‐use decisions and improving conservation outcomes and scope for restoration (Tengö et al., 2014). Broader definitions of expertise may be essential for developing models of ecosystem viability that are both ecologically and socially acceptable (Harrison et al., 1998).

Variation among experts

Surveys in which expertise is broadly defined are likely to result in diverging judgments but, if undertaken with structured methods, can provide data‐driven evidence of systematic differences in opinions between groups and individuals (Hemming et al., 2018). Such evidence can inform discussions with stakeholders and decision makers about cases in which opinions are shared and in which they diverge (e.g., Eyvindson et al., 2019). In the case of BGGW, most experts tended to agree that a site was viable when large trees were present and the ground layer was diverse and dominated by native species. Policies and regulations targeted toward the management and protection of these areas are likely to be broadly acceptable among the experts we surveyed. However, persistence of the BGGW ecosystem may require extensive restoration beyond existing highly functioning remnants (Tierney, 2022). This may necessitate targeting areas where the ecosystem has undergone an intermediate form of transformation (e.g., grazed woodlands and derived native grasslands [Spooner & Allcock, 2006]), for which disagreement was more prevalent. These vegetation states are also valued for their contribution to livestock grazing and potential for more intensive land uses (e.g., cropping [Carr‐Cornish & Hall, 2016]). Even if there were general support for targeting these areas for conservation and restoration, our results suggest that there may be very different views about the resources and actions required. Overall, experts from government conservation departments were generally optimistic, consistent with the view that these patches retain the capacity for natural regeneration (e.g., Chazdon et al., 2020). Others, including many experts from government agriculture, may believe these areas have little likelihood of sustaining defining features in the absence of intensive and costly interventions (e.g., Cuneo et al., 2018). Individuals from each of these groups play a range of roles in the formulation, communication, and implementation of policies and regulations that affect the BGGW ecosystem, and disagreements over whether examples of the ecosystem are viable suggest potential for different advice both in government and to land managers.

Differences in indicator coefficients among groups of experts may be indicative of alternative models of viability. Coefficients only differed in the case of relationships with non‐native plant cover and suggested a preference for lower non‐native plant cover among individuals from government conservation and with low management expertise. Non‐native invasive species are often thought to be an indicator of ecosystem degradation and a direct cause of ecosystem collapse (Newton et al., 2021). However, there are diverging opinions on whether non‐native species are bad or good and on whether they drive degradation (Estevez et al., 2015; Shackleton et al., 2022). In savannah woodland ecosystems, non‐native species have been intentionally introduced to improve forage for livestock, and associated land management practices have facilitated invasions (Murray & Phillips, 2012). Few, if any, patches of the BGGW ecosystem lack non‐native plant species (Prober et al., 2016). Although some non‐native species may have direct ecological consequences, others have neither known ecological impact nor economic value, yet they may be perceived negatively or positively by different individuals.

Professional affiliation and social group membership, often associated with shared norms and experience (Farrow et al., 2017), can affect opinions, preferences, and decision‐making in biodiversity conservation (Karns et al., 2018; Nordén et al., 2017). Although differences owing to affiliation could have important practical implications for ecosystem assessment, in our study much of the systematic variation among individuals remained unexplained. An expert's current affiliation is unlikely to provide a comprehensive measure of differences in norms and experience, and many individuals could have held a secondary position elsewhere or have worked for multiple organizations throughout their careers. Unconscious systematic bias in individual judgments is influenced by a wide diversity of motivational, sociodemographic, cognitive, and psychological factors (Kerr et al., 1996; Tam & McDaniels, 2013), and further research would be necessary to understand their role in perceptions of ecosystem viability.

Statistical models of collapse and viability drawing on structured expert elicitation still have potential to contribute to consistent and transparent ecosystem assessment. However, when judgments vary, average models may have less utility, but they could provide an opportunity to highlight cases where individuals or groups have shared or diverging opinions. In Australia, policies, guidelines, and resources are available for the conservation and recovery of BGGW, yet there is mixed evidence that these have been effective (Vardon et al., 2023). There continues to be competing priorities, primarily with agriculture (Heagney et al., 2021), but also increasingly with other sectors (e.g., renewable energy generation) (Foley, 2024). Finding solutions to manage these and support ecological recovery of remaining patches will be necessary for conservation and restoration of BGGW and threatened ecosystems more widely. Determining where differences in opinion arise among key groups of experts could inform discussion with decision makers and support identification of appropriate policy and management interventions for the conservation and restoration of threatened ecosystems.

OPEN RESEARCH BADGES

This article has earned Open Data and Open Materials badges. Data and materials are available at https://doi.org/10.6084/m9.figshare.23807199.

Supporting information

Supplementary Information.

Dorrough, J. , Travers, S. K. , Val, J. , Scott, M. L. , Moutou, C. J. , & Oliver, I. (2025). Evaluating models of expert judgment to inform assessment of ecosystem viability and collapse. Conservation Biology, 39, e14370. 10.1111/cobi.14370