Quantifying conditional probability tables in Bayesian networks: Bayesian regression for scenario-based encoding of elicited expert assessments on feral pig habitat

ABSTRACT

Bayesian networks are now widespread for modelling uncertain knowledge. They graph probabilistic relationships, which are quantified using conditional probability tables (CPTs). When empirical data are unavailable, experts may specify CPTs. Here we propose novel methodology for quantifying CPTs: a Bayesian statistical approach to both elicitation and encoding of expert-specified probabilities, in a way that acknowledges their uncertainty. We illustrate this new approach using a case study describing habitat most at risk from feral pigs. For complicated CPTs, it is difficult to elicit all scenarios (CPT entries). Like the CPT Calculator software program, we ask about a few scenarios (e.g. under a one-factor-at-a-time design) to reduce the experts' workload. Unlike CPT Calculator, we adopt a global rather than local regression to ‘fill out’ CPT entries. Unlike other methods for scenario-based elicitation for regression, we capture uncertainty about each probability in a sequence that explicitly controls biases and enhances interpretation. Furthermore, to utilize all elicited information, we introduce Bayesian rather than Classical generalised linear modelling (GLM). For large CPTs (e.g. >3 levels per parent) we show Bayesian GLM supports richer inference, particularly on interactions, even with few scenarios, providing more information regarding accuracy of encoding.

1. Introduction

In different fields, elicitation of expert assessments may prove to be an excellent portal to uncertainty assessment [61]. Understanding the sources of uncertainty not only helps improve reliability but it may also ensure a better quantitative analysis, with reduced risk [62]. This paper proposes a ‘structured’ elicitation approach [51] to quantify – with uncertainty – the conditional probability table (CPT) entries that define a Bayesian Network (BN). This type of model is often used in environmental risk assessments e.g. for understanding stressors of coral reefs [5] and water treatment infrastructure [68]. More generally, BNs are graphical models that have become a commonly used approach in the modelling of uncertain knowledge [24,66,67] and experiencing increased uptake in many fields of enquiry [30,54,71,81], particularly those relying on expert knowledge to replace or complement empirical data [46].

BNs represent the probablistic relationships between the input variables (parent nodes) and outcomes (child nodes) of a model. The relationships between these nodes are represented by arrows. These relationships can be quantified using conditional probability tables (CPTs) [24]. In our feral pigs example (detailed in Section 2) the adequacy of food supplies to support feral pig populations (child node) depends on three parent nodes: food quality, duration (how long the food source is available) and accessibility (access to food resources), as shown in Figure 1.

Figure 1.

BN sub-model of food resources (adequate or not for feral pigs), which depends on food quality (presence and nutritional value of food resources), duration (how long the source is available, e.g. continual year-round, seasonal) and accessibility (access to food resources).

Currently there exist several methods for quantifying these relationships among variables, expressed as CPTs in BNs. In some situations it is possible to ask the expert for an estimate of the correlation between pairs of variables [21,40,58,59,87]. These correlations can be elicited when nodes are continuous [21,58,59] or when nodes are ordinal [40,87]. Asking for correlations $\rho =\mathrm{Corr}(Y,X_1)$ [57, p. 3] mandates some careful preparation of experts [87], to calibrate their estimates to the appropriate range of values of Pearson's or Spearman's correlation (for continuous or ordinal variables, respectively). Of interest here is the situation where the CPT is large, which typically involves more than one parent node. We note that elicitation of correlations becomes more complex with multiple parents [87, p. 817], with nuanced questioning required to elicit marginal, conditional or joint correlations.

However, in this paper, we consider the situation where all nodes of BNs are ordinal or nominal, so that their joint probability distributions are quantified by CPTs, defined on a discrete domain [59]. CPTs can be used, even when variables are continuous, by categorizing variables. This can simplify the elicitation task, since experts can be asked about the probability of particular scenarios θ (rows in the CPT) rather than asking for more abstract information about the parameters governing a continuous representation of $p(\theta )$. We consider the general situation where experts are not very technically minded [87, p. 817] so that a general preference for discrete nodes can simplify elicitation, allowing scenario-based elicitation as discussed here.

When no empirical data are available, the CPT can be, and most commonly is, fully specified by experts (e.g. [71]). Such guidelines on eliciting CPTs generally advise modellers to simplify the elicitation task by keeping to a minimum number of parent nodes (defining the columns of the CPT) and parent levels (specifying the scenarios in each row of the CPT). However, the BN literature indicates that elicitation of more complex CPTs is often considered to be too demanding for the experts, because of the time requirement [5,30,54]. In this paper, we aim to ask the experts about a few scenarios to reduce their workload. Previous research, (e.g. [18,30,71]) has tended to employ the CPT Calculator, which ignores uncertainty. Following standard practice (perhaps due to widespread use of CPT Calculator software), we use a one-factor-at-a-time (OFAT) design for choosing scenarios as detailed in Section 3.2.1.

Whilst expert knowledge has long been used to populate CPTs [20,54,55], almost all of these approaches focus on eliciting point estimates of probabilities in the tables, i.e. ignoring the uncertainties. This deterministic approach overlooks the situation where experts are not certain of their probability assessments, and hence it is not apparent where their estimate sits within their mental view of plausible values $p(\theta )$. Are the expert's estimates conservative or optimistic (i.e. either under- or over-stating θ), or do they represent a ‘best’ estimate, such as the mean, median or mode? (Here best is referring to the expert's ‘best’ estimate, rather than an estimate best satisfying some mathematical criterion.) Thus, we aim to explicitly recover a probability distribution $p(\theta )$, which can explicitly represent both the experts' knowledge and their uncertainty about the CPT entries [32].

Thus, statistical approaches to scenario-based elicitation of CPTs requires that uncertainty about these probabilities is captured directly from the experts [51,72] for each scenario, by encoding the probability distribution $p(\theta )$. Recent studies of eliciting CPTs and uncertainty [5,30] have adopted the ‘4-point’ approach [72]. This 4-step process is also referred to as the ‘Inside-out’ approach since it starts by eliciting the ‘inside’ of the probability interval and then moves ‘outwards’ to elicit an interval. For this reason the 4-point (Inside-out) approach belongs in the Frequentist school so that when they ask the expert for their best estimate, they are actually obtaining a mean estimate $\hat{\theta }$, and then the bounds are interpreted as a $100(1-\alpha )$% confidence interval for $\hat{\theta }$, where:

$$\mathrm{Pr}[(L,U]\supset \hat{\theta })=1-\alpha .$$ (1)

Thus, the confidence interval for $\hat{\theta }$ is narrower than for θ itself, by a factor of $1/\sqrt{\nu }$, where ν is the effective sample size used to estimate θ. Also, these confidence intervals are typically estimated using asymptotic Normal approximations [72] and hence are necessarily symmetric. We note that an experts' uncertainty around a probability is likely to be skewed [64], which using this ‘Inside-out’ formulation can be mistakenly interpreted as ‘over-confidence’ [53,64].

The importance of eliciting extreme values before the expert's best estimate, though not necessarily recognized in ecology [16], has long been recognized in the risk assessment literature, where it is known as the Program Evaluation and Review Technique (PERT) for eliciting uncertainty around the probability of any scenario [15]; [62, chap. 6]. More recently the nomenclature ‘Outside-in’ was coined to explicitly differentiate the order of elicitation from methods that do not specify the order (e.g. the 4-point method), and hence are implicitly ‘Inside-out’ [72]. Proposed at a similar time (2009–2010), the ‘Outside-in’ method use similar questions to the 4-point (Inside-out) method, but in different order and subtly different statistical meaning. The ‘Outside-in’ reverses the order of elicitation in [72], starting with the ‘outside’ (L and U bounds) and moving towards the ‘inside’ (the expert's best estimate). Typically the exact sequencing of questions is not specified for PERT, however the tabular format implies a sequence and asks, from left to right, for a pessimistic value, their estimate of the most likely value, and then the optimistic value. As neither the confidence nor plausible interval interpretations are enforced, it will not be clear which one is applied by the expert.

Furthermore, constructing questions in the ‘outside-in’ way is more sympathetic to a Bayesian (rather than a Frequentist) interpretation: uncertainty refers to the plausible values of the probability θ rather than the precision of $\hat{\theta }$ [37,75,78]. This sequencing can avoid biases such as overconfidence, anchoring and adjustment [49,51,60,64]. Indeed, elicitation can be subject to a wide range of biases, as discussed in texts [62, Chapters 1–3] and reviews (e.g. [48,49,73,76]). We recommend elicitation is designed to minimize those biases of importance [52]. In particular, we propose the ‘Outside-in’ method using a Bayesian interpretation, as the basis for eliciting uncertainty on CPT entries; this is new in the context of eliciting CPTs in BNs.

Another challenge arises from the other scenarios that are not elicited. From the literature on BNs, the extensively used CPT Calculator adopts a deterministic method (i.e. without quantifying uncertainty) of encoding the CPT entries which were not elicited, namely linear interpolation (e.g. [5,18,30]). Linear interpolation becomes increasingly difficult as CPT tables become more complex. Thus it is not surprising that the CPT Calculator software is constrained for application to simple situations, i.e. where parent nodes have at most just three levels. Hence, for large CPTs (such as the CPT with 625 scenarios as shown in Section 2) that tool is not suitable for interpolating the remaining scenarios. Additionally, linear interpolation embedded within the CPT Calculator, can be viewed as a ‘local’ form of regression (with no uncertainty) because interpolating each scenario can rely on only a small number of scenarios elicited from the expert. This paper addresses these limitations by using a generalised linear model (GLM) as a ‘global’ form of regression, that moreover accounts for varying uncertainty about every scenario, as a second element of innovative contribution. Here, we extend existing methods for scenario-based elicitation, previously applied to regression [44], coincidentally in the context of modelling habitat suitability. This extension involves use of a Bayesian approach to obtain predictions of the remaining cell entries with uncertainty. Both elements, by combining Bayesian interpolation with Bayesian interpretation of elicited intervals (via ‘Inside-out’ elicitation), can be combined to define a ‘Bayesian Regression for Scenario-based Encoding’ (BRSE) which is new in scenario-based elicitation, including in the context of eliciting CPTs in BN.

This paper will illustrate this new Bayesian approach using a case study of a CPT in feral pig habitat suitability (Section 2). This provides a basis for explaining the methodology including eliciting probabilities with uncertainties and extrapolating the remaining scenarios embedded within BRSE (Section 3). Section 4 explains the results and main findings of this case study. Section 5 summarises our contributions, including a comparison among models.

2. Case study

Often, invasive mammalian wildlife animals can be harmful for the environment, economy, agriculture as well as transmitting diseases to human and animals [9,43]. To relieve and manage the impacts of such pests, there is a need to determine the spatial pattern of areas at risk related to habitat suitability for those animals [29,65,85]. For example, feral pigs (Sus scrofa) are one of the most common harmful wild mammals in the world [6]. In tropical northern Australia, invasive feral pigs are a significant potential threat to ecological systems because they cause damage to soil, plants and water bodies [14]. In addition, they carry diseases that threaten native wildlife and also humans, such as tuberculosis [10,47]. A first step in managing this feral pest is mapping areas at risk [81]. Trapping data is available but is patchy, hence here we describe part of the process of mapping areas at risk using expert knowledge. A BN provides a framework for incorporating this information to describe key drivers of its habitat requirements [54,71,81].

We chose an existing case study based on the findings from an expert elicitation workshop run for feral pig management in Western Australia. The variables and relationships represented in the BN were based on the model framework developed for feral pigs in northern Australia [30]. Additionally, a similar BN structure was used for the same species in the Northern Territory (central northern Australia) [27]. Justine Murray (JM) and Jens Froese (JF) conducted an expert elicitation session with fourteen wild pig experts, rangers from researchers to land managers and landholders, seeking their expert knowledge about habitat requirements for feral pigs. Together, the group proposed a set of factors considered as key drivers of feral pig habitat. These nodes were: Water, Food, Seclusion and Shelter with levels and definitions detailed in Appendix 1. The experts proposed a structure of relationships for these variables. Altogether, five CPTs were proposed to link together the five nodes as shown in [30].

The raw data elicited from the experts for all tables is contained in Appendices 4–8. The experts worked in small groups, one group per CPT. The largest CPT, for Habitat Suitability given Water, Food, Shelter and Seclusion contained $5\times 5\times 5\times 5=625$ cells. For larger CPTs, experts considered only some combinations of categorical variables. The experts were asked to provide lower and upper limits to the plausible values for each probability (θ), the plausibility, and then the expert's best estimate.

3. Methods

3.1. Probabilistic formulation of elicited assessments

Briefly, the graph that defines a BN comprises variables represented as a vector of parent nodes $X=(X_1,X_2,\dots ,X_K{)}^{\mathrm{T}}$, where K is the number of parent nodes, and the outcome (child node) Y. Arrows indicate the relationships among variables. BNs capture the expert knowledge about the conditional probabilities (CPs) $\theta =Pr(Y|X)$ that describe the relationship between the child node and its parent nodes [24]. For example, Figure 2 represents a very simple example of a sub-model in a BN from our case study of habitat suitability for feral pigs, with similar problem to that detailed in [30], for a different study region, different experts, different elicitation and encoding methods but similar elicitors. Here the relationship between the child node (water quality adequate for feral pigs, Y) depends on two parent nodes, which are the presence of water ($X_1$) and the salinity of water ($X_2$). Table 2 shows the corresponding CPT, where each row corresponds to a particular ‘scenario’, being specific values of parent and child nodes. Specifically:

$${\pi }_{ij}=Pr(Y=i|X=\mathrm{Parents}(Y)=j),$$ (2)

where ${\pi }_{ij}$ is the conditional probability that child node Y is in particular level i, given a particular combination of values of the parent nodes j, with $X_1=j_1,X_2=j_2,\dots ,X_K=j_K$. Here we consider binary outcome $i\in \mathrm{\Omega }=\{0,1\}$, however, more generally these results can be extended to an n-ary case where $i\in \mathrm{\Omega }=\{0,1,\dots ,n-1\}$, with n possible values. Here, $j_k$ is a particular level of the kth parent node with $j_k\in \{1,2,\dots ,m_k\}$ where $m_k$ is the number of levels for kth parent node, and $j=(j_1,j_2,\dots ,j_K{)}^{\mathrm{T}}$ is a particular combination of levels of all parent nodes and K is the number of parent nodes. Whilst j is a multivariate description of the values of parents in each scenario, we can define each scenario $s=\{1,2,\dots ,S\}$, by the combination of child value i and parent values j. So there is a unique one-to-one mapping with $\{Y=i,X=j\}\iff S=s$ for particular i, j and s. For simplicity we use ${\theta }_s\equiv {\pi }_{ij}$ in Equation (2). We allocate the label s = 1 to correspond to what the expert considers to be the ‘best’ scenario leading to the most probable outcome $\arg \underset{s\in S}{max}Pr(Y=i|S=s)=1$.

Figure 2.

BN sub-model of Water quality (adequate or not for feral pigs), depends on Presence of water with two levels (yes or no) and salinity of water with three levels (high, moderate or low).

3.2. Eliciting and encoding probabilities with uncertainty

Elicited assessments can be represented without uncertainty (see [7,39]). The first step in eliciting a CPT, is to decide which scenarios to elicit. Thus we use the most common approach is to follow a popular design implemented by the CPT Calculator tool (Section 3.2.1). The second step is to do decide how to elicit the probability of each scenario, including how to elicit the expert's uncertainty (Section 3.2.2). Then their assessments are encoded into statistical distributions (Section 3.2.3). Finally, it is necessary to extrapolate the remaining scenarios that were not elicited (Section 3.2.4).

3.2.1. CPT calculator

A common approach to eliciting CPTs chooses to elicit all scenarios, i.e. all rows of the CPT. Marcot et al. [54] make this approach feasible by advising modellers to keep their CPTs as small as possible. Another common approach is implemented by CPT Calculator, which is a software programming tool developed by Cain [18] to fill out CPTs from a small number of elicited scenarios [8,30,71]. To choose the scenarios, it starts with the best scenario where all parent factors have the highest (best) levels, meaning that the outcome i has the highest chance of occurring. Then scenarios are selected, which are almost the same as the best scenario, except that only one parent factor is not at the best level.

CPT Calculator uses the elicited scenarios (that are close to best) and applies linear interpolation to interpolate the scenarios that the expert did not have time to elicit. When comparing any scenario to the best one in terms of the difference, an interpolation factor (IF) is calculated [18, Appendix 2]. One major limitation of the way that the linear interpolation underlying CPT Calculator has been implemented is that each node is constrained to have just two or three levels. Hence, for large CPTs that have more than three levels, this tool is not suitable. In addition, with CPT Calculator's deterministic fashion of interpolation, it is not possible to incorporate additional information of the expert's uncertainty about ${\theta }_s$ without substantially changing the underlying calculations.

A recent example uses an extension of the CPT Calculator approach [30], by replacing estimation of simple point estimates ${\theta }_s$, combining it with the ‘Inside-out’ method [72] to elicit confidence intervals around ${\hat{\theta }}_s$. That study relied on linear interpolation in CPT Calculator to estimate the remaining entries in CPTs having factors of three levels or less. However, for a larger CPT, that study used Fenton's method [27] for interpolation, as implemented in AgenaRisk v.6.1 software [1]. One disadvantage of this approach is that the uncertainty was represented by confidence in the expert's best estimate $\widehat{{\theta }_s}$ as detailed in Equation (1). Instead, we find it advantageous to elicit the expert's plausible range of values for the whole range of ${\theta }_s$, not just ${\hat{\theta }}_s$. Thus, this study will address these limitations by investigating using scenario-based elicitation method as discussed below.

3.2.2. Outside-in method (reversing the 4-point process)

The ‘Outside-in’ approach [51] is so-named because (like PERT) it starts by asking the first two questions (Appendix 2) to elicit the extreme or ‘outside’ values, establishing lower L and upper U bounds such that ${\theta }_s$ falls between them. The third question seeks the experts' assessment of the plausibility P that ${\theta }_s$ falls between bounds L and U, such that:

$$Pr(L\le \theta \le U)=P.$$ (3)

The last question elicits the expert's best estimate of ${\theta }_s$, as the mode M of the expert's distribution of uncertainty $p(\theta )$ around θ, which is the most plausible value of ${\theta }_s$ (for that scenario) where:

$$M=\arg \underset{\theta }{max}p(\theta )$$ (4)

(For precise wordings of questions suitable for the case study, see Appendix 2.) The ‘Outside-in’ interpretation, which through explicit sequencing, aligns with Bayesian focus on the plausible range of values for the probability itself [75]. In contrast to the ‘Outside-in’ method, the ‘Inside-out’ approach reverses this order of questions, starting with question 4, and then progresses to questions 3, 2 and 1. As discussed in Low Choy et al. [51], this reflects Frequentist focus on uncertainty the expert's best estimate. The ‘Outside-in’ approach has subtle though notable differences with the well-established PERT procedure. With PERT, the absolute extreme rather than probabilistic lower and upper bounds are elicited, and the sequencing of questions implicitly progresses from pessimistic to most likely (mode M) to optimistic time of a task. Although this results in an uncertainty interval around the probability θ, it does not fully ‘zoom out’ the expert's thinking to the extremes before focussing on their best estimate.

3.2.3. Encoding probabilities with uncertainty

An encoding method aims to translate elicited information from expert knowledge into statistical distributions reflecting their best estimates and uncertainties. Whilst focusing on a particular scenario s, U and L were elicited as the upper and lower limits (the narrowest interval for which $Pr(L\le \theta \le U)=P$ is still true). To be a bit more realistic, the experts' will not choose the bounds, L and U, to be 0 and 1 respectively if they wish to be informative (in contrast with PERT). Furthermore, it is possible that there are several intervals [L, U] with $Pr(L\le \theta \le U)=100\mathrm{\%}$, and hence we need to make sure that the expert uniquely defines this interval as the smallest interval such that $Pr(L<\theta <U)\approx P$.

Eliciting P helps describe the experts' uncertainty about θ. When unavailable, we may alternatively specify the weight of prior evidence ν [62, chap. 6].

To describe uncertainty in θ we allocate an adjusted form of a Beta distribution. First define ${\theta }^{\ast }\sim \mathrm{Be}(a,b)$, which gives positive probability to any proportion θ except at the endpoints of the unit interval, so $0<{\theta }^{\ast }<1$. We note that the values 0 and 1 are not strictly ‘possible’ under this choice of a Beta distribution, since the log odds of these are not defined, in the elicitation Equation (5) nor the regression Equation (9). In this case, the experts specified lower and upper endpoints that fell inside the unit interval, $[L,U]$. We can constrain the positive probability to occur within this reduced interval simply by ‘shrinking’ the support of the distribution (of unconstrained ${\theta }^{\ast }$) in proportion to the interval $[L,U]$ via:

$$\begin{array}{rl}\theta & =({\theta }^{\ast }-L)/(U-L)\mathrm{where}{\theta }^{\ast }\sim Be(a,b)\\ a+b& =\nu \mathrm{for}0<{\theta }^{\ast }<1\mathrm{so}\mathrm{that}L<\theta <U\end{array}$$ (5)

Hence a and b are shape and scale parameters which now reflect the expert's beliefs about uncertainty θ as represented in a Beta distribution. In previous studies a and b were estimated numerically, by interpreting L and U as elicited quantiles, and equating them to theoretical quantiles [53]. Here we instead interpret L and U as constraints on the distribution (Equation (5)), so there is zero plausibility of a value outside the interval. We can then follow the PERT procedure: to estimate a and b by specifying ν, the effective sample size or weight of expert knowledge. The choice of ν is explained below.

The effective sample size ν reflects the weight of the expert's knowledge, and can reflect a conservative or uncertain expert when set to a low value, e.g. 3, 5 or 10. The reason is that when a = b = 1 and $\nu =2$, then the distribution of uncertainty about ${\theta }_s$ would be completely flat (non-informative) and hence the most plausible value of θ is not defined. Furthermore, when $\nu <1$, then the most plausible values are close to 1 or 0. So, in many practical contexts these values are not meaningful. Here, as the expert did not consider any scenarios to have probability of habitat suitability for feral pigs to be both ‘either the lowest or the highest’ – it made more sense to be able to specify either a low or high (or a middle) value as being the mode (or most plausible) value. Therefore, we typically choose the effective sample size to be the lowest integer value $\nu =3$ above the non-informative case $\nu =2$, leading to a slightly more informative prior than the flat distribution. It would be possible in other situations to choose a more strongly informative prior with $\nu >3$. The expert declined to specify the effective sample size ν for each scenario. So we presume that the uncertainty is constant across all scenarios ${\nu }_s=\nu =3$, and set it to the most conservative value.

In previous studies, the formulation of scenario-based elicitation and encoding [44,51] allowed the experts to have different levels of uncertainty $P_s$ or ${\nu }_s$ in each scenario, rather than fixed uncertainty across all scenarios (as in [13]). Here we presume uncertainty is constant across all scenarios, that is ${\nu }_s=\nu ,\mathrm{\forall }s$ since we are given no information to the contrary. This seems to be a pragmatic approach when eliciting expert knowledge about many CPTs (and hence many θs), especially when eliciting a BN. The extra elicitation load of specifying L, U and M for every scenario already exceeds the usual elicitation load of just specifying $\hat{\theta }$, here encoded as the mode of the distribution of uncertainty, M, as in Cain [18] and Marcot [54].

So, returning to the problem here of encoding a and b given elicited information L, U, M and modelled assumption ν, simply following the method-of-moments as a robust form, which is also known as the PERT encoding scheme [[15], [62, chap. 6]], we can equate the theoretical (m) and elicited modes (M) such that:

$$m=(a-1)/(\nu -2).$$ (6)

Then, equating $m\equiv M$ can give:

$$\hat{a}=M\times (\nu -2)+1;\hat{b}=\nu -\hat{a}.$$ (7)

3.2.4. Extrapolating other CPT entries using Binomial regression

Then we can relate the effective number of successes, $a_s$ to the scenarios' characteristics, via a Beta regression. Denote by s the scenario where Y = i and X = j.

$$\begin{array}{rl}& {\theta }_s\sim \text{Beta}(a_s,b_s)\text{where }{\theta }_s=a_s/(a_s+b_s)\end{array}$$ (8)

$$\begin{array}{rl}& \text{logit}({\theta }_s)={\beta }_0+\sum _{k=1}^K\sum _{j_k=1}^{m_k}{\beta }_{j_k}\mathcal{I}[X_{ks}=j_k],\end{array}$$ (9)

where $\mathcal{I}$ is an indicator function defined by

$$\mathcal{I}[X=j]=\{\begin{array}{ll}1,& \mathrm{if}X=j\\ 0,& \mathrm{otherwise}\end{array}$$ (10)

and ${\beta }_{j_k}$ is the coefficient regression of $j_k$th level of kth factor X and $X_{ks}$ is the value of the kth habitat factor in scenario s. Since $X_{ks}$ are effectively split into ‘dummy’ variables $\mathcal{I}[X_{ks}=j_k]$, there needs to be an effect for each habitat factor k, at each possible categorical level j, denoted ${\beta }_{kj}$, e.g. ${\beta }_{12}$ is the 1st level of the 2nd parent, which in the feral pigs example is low salinity of water in Table 1. Almost all commercial software packages do not allow a Beta regression to use the additional information on expert uncertainty in θ, as provided here by $\nu =3$, or by ${\nu }_s$. Nor do these packages permit ${\nu }_s$ to vary across scenarios s. Here we can re-express the Beta regression as a kind of binomial regression to allow inference about the ${\theta }_s$, as the probability of presence. Following Low Choy [51] and James et al. [44], we change perspective and consider $a_s$ to be the random variable instead of ${\theta }_s$, with fixed ${\nu }_s=\nu$. Thus, the raw data is $A_s=M_s{\nu }_s$ (the effective number of presences from a sample of ${\nu }_s$ sites with this scenario). Here, ${\nu }_s=\nu$ is a constant, but may not be in other situations. This allows us to use a binomial regression allowing inference about how the habitat suitability factors impact on the expected proportion of sites with feral pigs being present ${\theta }_s$, and hence the expected number of such sites $A_s$ in a minimal sample of size $\nu =3$. Alternatively $a_s$ and $b_s$ can be encoded from $L_s$ and $U_s$, should these be elicited as credible limits rather than the hard limits considered here [51]. Therefore,

$$A_s\sim \text{Bin}({\nu }_s,{\theta }_s)$$ (11)

replaces Equation (8). Thus in general, the objective is to analysis the CPT entries, ${\theta }_s$, where scenario s denotes a particular outcome Y (child node) conditional on its parent nodes X set (combination of parent node values). The experts are required to the $L_s$ and $U_s$ bounds, the plausibility $P_s$ and the expert's best estimates $M_s$.

Table 1. CPT of water quality adequate (q) for feral pigs contained only six scenarios, defined by the six combinations obtained from two levels of water presence $X_1$, i.e. No (n) or Yes (y) and three levels of water salinity $X_2$, i.e. High (h), Moderate (m) or Low (l). In Equation (2), scenario # 1 gives the conditional probability of adequate water quality where the presence of water is yes and the water salinity is low, a value of 0.90.

Scenarios	Presence	Salinity	adequate
1	Yes	Low	${\theta }_{qyl}=0.90$
2	Yes	Mod	${\theta }_{qym}=0.60$
3	Yes	High	${\theta }_{qyh}=0.10$
4	No	Low	${\theta }_{qnl}=0.05$
5	No	Mod	${\theta }_{qnm}=0.05$
6	No	High	${\theta }_{qnh}=0.05$

Here the likelihood for the Binomial regression defined by Equations (9) and (11) is:

$$Pr(A_s|{\nu }_s,{\theta }_s)=\prod _{s=1}^{S}Pr(A_s|{\nu }_s,{\theta }_s)=\prod _{s=1}^{S}Pr(A_s|{\nu }_s,{\beta }_{j_k},X_{ks}).$$ (12)

In practice, the elicitors presumed that the expert assessments of all scenarios should be conditionally independent given the factors $X_{ks}$, which would be strictly fulfilled if all scenarios are elicited separately. In this case these scenarios were elicited at one elicitation session. Therefore, it is possible that scenarios could be elicited with some carryover effects between consecutive scenarios. However, we assume this kind of correlation is negligible, and that elicitations are referring mostly to the scenario of interest. In reality, however, many experts will ensure that their elicitations are ‘coherent’ and hence will compare their elicitations across scenarios. Thus we make the much weaker assumption that elicitations are exchangeable, conditional on knowing the factors defining that scenario. In practice, we assume that beyond this dependence on the factors defining the scenarios, there is negligible correlation amongst elicitations.

We interpolate the remaining CPT entries by predicting the response for each scenario, together with a posterior standard error. Formally, the logit transformation can be written as: $\text{logit}({\theta }_s)=\log ({\theta }_s/(1-{\theta }_s))$. Therefore, solving for ${\theta }_s$ gives:

$${\theta }_s=\frac{e^{{\eta }_s}}{1+e^{{\eta }_s}}=\frac{1}{1+e^{-({\eta }_s)}}.$$ (13)

For clarity Equation (13) has been applied manually (yielding the same results for classical and Bayesian GLM) for the food adequacy CPT (Table 3, scenario number 22). Using the logistic regression model, we predict:

$${\hat{\theta }}_{fhlm}=\frac{1}{1+e^{-(0.98-0.53-0.12)}}=\frac{1}{1.72}\approx 0.58,$$ (14)

where the intercept ${\beta }_0=0.98$, the effect estimate size of duration in the low level is $\widehat{{\beta }_2}=-0.53$ and the effect estimate size of accessibility in the moderate level is $\widehat{{\beta }_3}=-0.12$ as shown in Table 2. This result is different to the value of 0.46 obtained via linear interpolation.

Table 3. The predictions of food CPT for Bayesian, Classical GLMs and CPT Calculator, and elicited values.

s	Quality	Duration	Accessibility	Expert's Best estimate	Bayes GLM predictions	GLM predictions	CPT Calculator predictions	Bayes GLM SEs	GLM SEs
1	High	High	Easy	0.90	0.73	0.77	0.90	0.09	0.11
2	Low	High	Easy	0.23	0.29	0.28	0.23	0.11	0.12
3	Moderate	High	Easy	0.70	0.70	0.70	0.70	0.13	0.14
4	High	Low	Easy	0.58	0.61	0.61	0.58	0.10	0.11
5	Low	Low	Easy	–	0.19	0.16	0.17	0.10	0.11
6	Moderate	Low	Easy	–	0.58	0.53	0.46	0.19	0.24
7	High	Moderate	Easy	0.80	0.78	0.80	0.80	0.16	0.18
8	Low	Moderate	Easy	–	0.36	0.32	0.21	0.24	0.29
9	moderate	Moderate	Easy	–	0.76	0.74	0.62	0.20	0.28
10	High	High	Hard	0.32	0.35	0.35	0.32	0.10	0.10
11	Low	High	Hard	–	0.08	0.08	0.12	0.05	0.05
12	Moderate	High	Hard	–	0.33	0.27	0.26	0.28	0.19
13	High	Low	Hard	–	0.24	0.24	0.23	0.12	0.12
14	Low	Low	Hard	0.08	0.05	0.05	0.08	0.05	0.03
15	Moderate	Low	Hard	–	0.22	0.15	0.19	0.19	0.18
16	High	Moderate	Hard	–	0.43	0.39	0.29	0.24	0.31
17	Low	Moderate	Hard	–	0.10	0.07	0.12	0.18	0.11
18	Moderate	Moderate	Hard	–	0.40	0.31	0.24	0.28	0.38
19	High	High	Moderate	0.70	0.70	0.70	0.70	0.13	0.14
20	Low	High	Moderate	–	0.27	0.21	0.19	0.17	0.17
21	Moderate	High	Moderate	–	0.68	0.62	0.55	0.20	0.27
22	High	Low	Moderate	–	0.58	0.53	0.46	0.19	0.24
23	Low	Low	Moderate	–	0.18	0.12	0.15	0.17	0.14
24	Moderate	Low	Moderate	–	0.55	0.44	0.37	0.25	0.38
25	High	Moderate	Moderate	–	0.76	0.74	0.62	0.20	0.28
26	Low	Moderate	Moderate	–	0.33	0.25	0.18	0.27	0.33
27	Moderate	Moderate	Moderate	–	0.74	0.66	0.49	0.25	0.43

Table 2. Comparison of Food CPT coefficient estimates (coef.est) and coefficient standard errors (coef.se) between Bayesian and Classical GLM.

	Bayesian GLM		Classical GLM
	coef.est	coef.se	coef.est	coef.se
Intercept	0.98	0.46	1.20	0.59
Quality_Low	−1.87	0.63	−2.14	0.75
Quality_Mod	−0.12	0.77	−0.36	0.91
Duration_Low	−0.53	0.57	−0.74	0.70
Duration_Mod	0.30	1.01	0.18	1.27
Accessibility_Hard	−1.58	0.57	−1.83	0.69
Accessibility_Mod	−0.12	0.77	−0.36	0.91

3.2.5. Bayesian inference

Bayesian statistical modelling can be useful since it offers a flexible interpretation of probability beyond frequencies to encompass hypothetical frequencies in ‘thought’ experiments or parallel universes, and importantly for risk assessment, the ‘degree of belief’ in an outcome [45,63]. Previously scenario-based elicitation for inference about regression coefficients was introduced in [44,51] with uncertainty on Y given X. Similarly point-of-truth modelling presented in [7] also used scenario-based elicitation but without uncertainty. Both approaches have utilised classical (Frequentist) methods for estimating regression coefficients that define the expert's mental model (Equation (9)). However, a Bayesian setting is useful since it may be applied within a philosophy that all probabilities $p(Y_s|X)$ can be interpreted as degrees of belief [50]. In addition, Bayesian approaches are useful in situations like scenario-based elicitation where the amount of data S is small, which applies here, since the number of scenarios (S) is between 5 and 18. In addition, the Bayesian GLM may flexibly adjust for expert assessments of interactions among factors, and in particular, the Bayesian approach ensures that all interactions are estimable, and reflect a non-informative prior when little information is elicited from experts on those interactions. The expert's underlying ‘mental model’ will be encoded using a Bayesian regression with priors (i.e. what we know before looking at the elicited data) which are non-informative, since explicitly we know nothing beforehand about effect sizes [11,34,36].

Mathematically, by using Bayes theorem Bayesian regression aims to estimate the posterior distribution $p(\beta |X,A)$ of the regression coefficients β based on the data A and X:

$$p(\beta |X,A)\propto p(A|X,\beta )p(\beta ).$$ (15)

Here $p(A|X,\beta )$ is the likelihood of elicited data A given X and effect size estimates, and $p(\beta )$ is the prior of the regression coefficients β.

For each β we are presuming a Student-t [34] prior distribution with a mean of zero, scale parameter of 2.5 and one degree of freedom; this is a Cauchy so that $\beta \sim t_1(0,2.5)$. As carefully described in Gelman et al. [34] the reason is to provide a minimal informative prior, in order to avoid problems of unstable estimates that could appear with completely non-informative prior or a vague $N(0,{\sigma }^2)$ specified such that the variance ${\sigma }^2$ is very large.

In the R programming environment [69], we use the arm package (for data Analysis using Regression and Multilevel modelling, as introduced in [35]) with the function bayesglm in order to analyse the elicited information using Bayesian GLM. Equivalent functions exist in the newer packages brms [17] and rstanarm [38]. The latter packages can provide Markov Chain Monte Carlo (MCMC) simulations from the fitted joint posterior distribution of all parameters

$$\{{\beta }^{(t)},t=1,\dots ,T\}\sim p(\beta |X,A).$$ (16)

The bayesglm function permits the modellers to estimate the effect sizes $\beta$ providing standard errors as a summary from $p(\beta |A,X)$. In a logistic regression (Equation (9)), we use the logit link function, although others are supported (probit and Cauchit). In addition, the remaining missing scenarios that were not elicited can be interpolated. See an example of code in supplementary material in Appendix 3.

3.2.6. Bayesian model fit criteria

It is important to evaluate models fit by Bayesian GLM. In particular, the arm package provides several model fit statistics including the Akaike Information Criterion (AIC) [2] and Deviance Information Criterion (DIC) [74]. However, in Bayesian inference, DIC has some limitations arising from not being fully Bayesian (i.e. it relies on a point estimate) [80]. Instead there are other options including posterior predictive checks (PPCs) [31,33], the Leave-One-Out cross-validation (LOO) as illustrated in Vehtari et al. [82] and the widely applicable information criterion (WAIC) in [86]. The PPCs can be used to compare between the actual (elicited) data $A_s$ and data ${\tilde{A}}_s$ simulated from the posterior predictive distribution. The LOO is a kind of cross validation which reruns analysis, each time leaving one scenario s out, while WAIC provides an alternative to DIC in estimating the expected log predictive density. Both WAIC and LOO are recommended when the available datasets S is large [83, pp. 20–21]. However, when S is small, as in our case, these measures are not suitable [17, p. 12]. Hence, in this paper we use PPCs to assess and compare the Bayesian model fits. Here we will use the brms (Bayesian regression Models, as introduced in [17]) package with function brm to assess the model fits.

3.2.7. Bayesian posterior predictive checks

With Bayesian modelling, posterior predictive checks (PPCs) can be used to assess and compare model fits, and are particularly useful when sample size is small [31]. This can be achieved by simulating data from the posterior predictive distribution $p({\tilde{A}}_s|A_s)$ to enable a comparison of elicited data $A_s$ to simulated data ${\tilde{A}}_s$. This requires integration [31] over the full range of unknown model parameters β:

$$p(\tilde{A}|A,X)=\int p({\tilde{A}}_s|\beta ,X)p(\beta |A,X)\mathrm{d}\beta ,$$ (17)

where $p(\beta |A,X)$ is the posterior distribution of β as shown in Equation (15). For each simulation from the posterior distribution ${\beta }^t$ (Equation (16)), we can generate the new (simulated) data from $p(\tilde{A}|A,X)$ by simulating from the data model ${\tilde{A}}^t\sim p(\tilde{A}|\beta ,X)$ conditional on parameters ${\beta }^t$. When the distribution of simulated data are similar to the distribution of elicited data, the model is fit very well and vice versa. In the R statistical programming environment [69], we will use the pp_check function in the brms package to implement PPCs.

3.3. Pooling posterior distributions across experts

There is a challenge when dealing with more than one expert [3,22]. The first question to examine is whether the differences can be meaningfully reconciled. If not, then aggregation would not be appropriate [62]. Otherwise, either behavioural or mathematical approaches may be used to pool multiple opinions [22,62]. For expert-defined BNs like in this feral pigs study, behavioural pooling was used to aggregate by consensus of opinions across experts in a group, and record their consensus as a single assessment of each ${\theta }_s$. The water quality CPT is an example. In contrast, mathematical pooling would combine estimates of a probability, each elicited from different experts, to find one aggregated the expert's best estimate and uncertainty. Here we may apply this approach to pool the Habitat CPT, which was separately elicited from three different experts as shown in Appendix 8 [62]. As used here, in linear pooling [56], opinions from many experts can be calculated by using an arithmetic average of the conditional probabilities from different experts as follows:

$${\overline{\theta }}_s=\sum _{\ell =1}^L{w}_{\ell }{\theta }_s^{\ell },$$ (18)

where $w_{\ell }$ is a weight given to the Lth expert and $\sum _{\ell =1}^L{w}_{\ell }=1$. Caley et al. [19] suggested that where more than one expert is available to provide opinions on a particular item, their opinions and the uncertainty around them might be weighted by ratings of expertise. We presumed equal weighting of each expert so that $w_{\ell }=1/L$ and hence, ${\overline{\theta }}_s=\sum _{\ell =1}^{L}1/L\times {\theta }_s^{\ell }$. For our case study, an equal weighted linear pooling is sufficient because all three experts have the same weight. An alternative approach uses p-values from training exercises to reweigh experts: this is Cooke's method of unequal weighted linear pooling [23,62]. Moreover, if the probabilities were very close to 0 or 1, then we could also consider geometric pooling [25]. In this situation we did not encounter too many situations like this, so linear pooling was deemed adequate.

4. Results

The model of all five tables in this paper was fitted by Bayesian GLM. In this section, we will examine two of these tables as examples, which are the availability of food table and then extend this to include more complex model of the habitat suitability CPTs from three experts. The rest of tables will be examined in Appendices 5–7.

4.1. Food CPT

Using Bayesian GLMs, the results of Food CPT (see left side of Figure 3) showed that all factors have effect sizes (coefficient values) plausibly far from zero. Having lower quality, lower duration and hard accessibility of food (i.e. in the worst case) will lead to the largest decreases in the chance that food is available and hence affects the potential of habitat suitable for feral pigs. On the other hand, in the best scenario (baseline), with high quality, high duration and easy accessibility of food, this leads to the highest probability that feral pigs will find food. We considered whether any interactions occurred, i.e. whether any of these factors amplified or dampened the effect of another. Here, the results demonstrated that the zero was the most plausible value for each of the interactions (involving two variables). Hence, there were no discernible interactions that could be estimated from this minimal set of scenarios.

Figure 3.

Bayesian GLM $95\mathrm{\%}$ credible interval (left side) and Classical GLM $95\mathrm{\%}$ confidence interval (right side) of Food CPT for only model (Solid line) versus main effects with all 2-way interactions (dashed line).

However, three of these interactions were centred slightly above zero. Interestingly, these all involved the lowest levels of these factors, namely: low quality of food, hard accessibility and low duration. Therefore, if further scenarios regarding these levels were to be asked of experts, then it would seem worthwhile to ask scenarios that provided information about these (i.e. allowing contrasts with low values of these factors). In addition, we found these three interactions all strangely have positive effects although they are all involved the lowest levels. This could be indicating that it is necessary to adjust upwards, any estimate of the chance of feral pigs, when relying solely on the main effects (i.e. corresponding to low values for food quality, accessibility and duration).

Comparing Bayesian GLMs and classical GLMs (Table 2) for the main effects model, there are differences in terms of the effect size estimates and their standard errors. Although the effect size estimates for Classical GLM are slightly larger than for Bayesian GLM, the credible intervals are narrower than the confidence intervals for the Classical approach, suggesting less uncertainty. For example, the width of the $95\mathrm{\%}$ credible intervals for the interaction between Quality and Accessibility of Food, in the worst case, for Bayesian GLMs is approximately 7 units, while in classical GLMs the width is approximately 11 units. Similarly, this applies to 7 out of 8 credible intervals. In addition, we found that both the Bayesian and Classical GLMs (Table 3, Figure 3) provided almost perfect fit to the expert elicited values. This can be explained because the model is saturated, which means that it is using all degrees of freedom as the number of parameters is almost the same as the number of data points. However, in the context of the model with main effects as well as all 2-way interactions, there are several important differences. All interactions are estimable via Bayesian GLMs, but only one out of 12 can be estimated for classical GLMs. In addition, the effect sizes of interactions are centred close to zero for Bayesian GLMs, while the only estimable interaction via classical GLMs is quite large when considering the worst case (i.e. Low quality, Low duration and Hard accessibility) and the remaining 2-way interactions are non-identifiable. In addition, the PPC diagnostics from Bayesian GLM provide richer feedback than the residuals from Classical GLM [84, Ch. 7]. Overall using Bayesian GLMs provides richer estimates.

In addition, Figure 4 illustrates that there are slight differences between the predictions from Bayesian GLM and CPT Calculator. For example, the best scenario is HHE (HHE refers to high quality of food, high duration and easy accessibility) and presents the largest difference of around 0.13. This is to be expected. CPT uses a deterministic method, centred on the best scenario, and interpolates (linearly) from scenarios that are very close to the best scenario. In contrast, the GLM approaches allows experts some leeway, and information from these scenarios is averaged in a non-linear way, that is spread across scenarios.

Figure 4.

For each scenario of Food quality, Food duration of availability and accessibility: Comparison between predictions of CPT Calculator and predictions of Bayesian GLM, where the letters H, L, M indicate High, Low or Moderate levels of food quality and duration, whereas E and H symbolise Easy and Hard levels of accessibility respectively, e.g. HHE refers to high quality, high duration and easy accessibility. The plus symbols refer to elicited values from experts and dots refer to the predictions that are not elicited, and instead are interpolated from the elicited scenarios.

Finally, Figure 5 shows the interpretation of model fit for the Food CPT. The distribution of the elicited (actual) data y are compared to the distributions of data simulated $y_{\mathrm{rep}}$ using 100 simulations from the posterior distribution of the parameters. Here, the elicited data follow a similar distribution to that of most simulated datasets, although a small proportion of the latter have more values towards the center (i.e. 5–10). Hence, we conclude that the Bayesian GLM adequately fits the elicited data that was used to estimate it.

Figure 5.

Model fit criteria for Food CPT where y refers to the elicited data (dark density) and $y_{\mathrm{rep}}$ refers to the data simulated.

4.2. Habitat suitability CPT

We will show the result for complex CPT, here for habitat suitability with parent nodes: water, food, seclusion and shelter. Complexity is in terms of size and pooling. Regarding size, this CPT has 625 possible entries, but only 18 scenarios were elicited. Pooling was required since three experts elicited those scenarios separately. Using the method described in Section 3.3, we pooled the habitat suitability CPT that was separately elicited from three different experts. Figure 6 showed the results of separated three experts and as well as pooling. It can be seen that all main effects of habitat suitability have effect sizes (coefficient values) plausibly far from zero. Having very good quality of water, seclusion, shelter and food (i.e. in the best scenario) will lead to the highest probability of habitat suitability for feral pig presence, since their coefficients are larger in magnitude. Furthermore, it is obvious that all main effects provided by the second expert groups (green lines) lead to a higher probability of finding feral pigs compared to the other expert groups. In addition, we noted that the effect sizes from the second expert group are closer to each other compared to others. On the other hand, all main effects in the worst case (i.e. the intercept) plausibly will lead to the largest decreases for the probability of suitable habitat.

Figure 6.

Habitat suitability CPT for main effects model results with probabilities θ pooled across expert groups.

For the interaction model, the results (see Figures 7 and 8) indicated that some of the interactions between two main effects have effect sizes (coefficient values) measurably close to zero. This implies that zero was close to the most plausible value for each of the interactions. However, there is only one interaction with plausible values far from zero: between shelter and seclusion, encoded from the second expert (green line), and has a quite positive effect size. Overall, the interaction model has over-fit the data, so we revert to the main effects model [41,70].

Figure 7.

Habitat suitability CPT for interaction model results with probabilities θ pooled across expert groups.

Figure 8.

Habitat suitability CPT intercept for interaction model results with probabilities θ pooled across expert groups.

Overall, Figure 9 shows that the quality of water and food availability are the most important factors as they have the most influence, and that seclusion and shelter had slightly less influence on the habitat suitability of feral pigs. For example, in the best scenario with very good water quality and abundant food, the probability of presence of feral pigs is high (close to red) only when both shelter and seclusion are present at very good level (the probability is approximately is $98\mathrm{\%}$). This aligned with biological knowledge of the team's ecologist. When water quality or food availability factors are reduced to a moderate level, the habitat suitability of feral pigs is lower (i.e. the probability of presence is between 60% and 65%). In addition, when only the shelter or seclusion factors is reduced to a moderate level, the habitat suitability of feral pigs is lower (i.e. the probability is between 75% and 80%). Similar pattern arises when two factors, shelter and seclusion, are reduced to a ‘good’ level (probability between 60% and 80%).

Figure 9.

Habitat Suitability CPT: main effects encoded from pooled experts. The bottom sections comprise sets of $3\times 3$ plots, showing the predictions with 3 levels in the response (of habitat suitability) using (right) Bayesian GLM and (left) CPT Calculator respectively. The top section of $5\times 5$ plots shows the predictions using Bayesian GLM with 5 levels (in the response). For instance, the bottom row contains 5 heat-maps where water quality is very poor and the left column has 5 heat-maps where food availability is very poor. In each heat-map plot: on the horizontal axis, there is a shelter variable varying from 1 on the left (very poor) to 5 on the right (very good); and on the vertical axis, the seclusion variable varies from 1 on the bottom (very poor) to 5 on the top (very good).

However, CPT Calculator cannot estimate the influence of habitat suitability effects because it only allows up to three levels for each factor. We can compare the CPT Calculator to our Bayesian encoding method, by choosing three levels, instead of five, for each of the parents i.e. high level (very good = 5), (moderate = 3) and low level (very poor = 1). This permits a comparison between both Bayesian GLM and CPT Calculator results (see Figure 9). As a result, we found that Bayesian GLM provides more details about the probability of prediction than the CPT Calculator in terms of more levels for parents. For example, when encoding the worst case scenarios where all factors had low levels using the Bayesian GLM (see Figure 9, right), the probability of the habitat being suitable for feral pigs was approximately zero, which matched the value elicited from the experts. In contrast, when encoding the worst scenario using CPT Calculator (see Figure 9, left), we found that the probability of habitat being suitable for feral pigs was larger than the expert-elicited value.

5. Discussion

The method we present for partially eliciting CPTs (with uncertainty) may be applied when only a few scenarios have been elicited, for example using the OFAT design, which is a standard practice used in CPT Calculator software. When only a few scenarios are elicited, there is time to ask experts further questions to discern their uncertainty regarding the probability of the outcome in each scenario (${\theta }_s$). This uncertainty flows through to the predictions, and was captured in the CPTs. This differs from the usual practice of producing tables solely filled with point estimates (as recommended by [54]). We suggest that, it is important to elicit the experts' uncertainty appropriately to gain accurate elicitation, particularly when their uncertainty about the probabilities is strongly skewed, e.g. when their best estimate is the same as L or U values [64]. Eliciting solely a point estimate, without the distribution of expert uncertainty across plausible values, means that the point estimate may not be the expert's best estimate, i.e. a mode (their most plausible value), median (50% plausibility that the probability is below this value), or mean. Instead, our practical experience has confirmed that the point estimate could instead lie anywhere between or on the edges of the plausible interval from L to U.

Also, when designing an elicitation, it is important to identify summary statistics to be elicited, including their order of elicitation. Identifying meaningful quantities to the expert is important, specially if experts have limited knowledge of statistics and probability theory [49]. For Bayesian networks, there is a long history of asking experts about all probabilities in a CPT, so that asking them a bit more about far fewer scenarios is not too large a departure from standard practice. In contrast, it may be possible to ask for correlations defining CPTs, but this will only be useful in situations where monotonically increasing/decreasing relationships define the CPT. In some situations, there may be substantial overheads in training and calibrating experts in what numerical values of a correlation represent.

For large CPTs, this study developed a new statistical modelling approach (Bayesian GLM regression) in order to interpolate the remaining cells of CPTs. A parallel project [30] involving the co-author JM chose to encode the elicited probabilities using uncertainty, but encoded these using the ‘Inside-out’ method instead of ‘Outside-in’ method; the latter allows a Bayesian interpretation which we feel is more aligned with the questions asked of experts, on the plausible values for each probability. Another major difference was the estimation procedure: they relied on linear interpolation [18] to estimate the remaining entries in the CPT. In addition, in this paper we explicitly compare results obtained using the three inference methods: Bayesian GLM, Classical GLM and linear interpolation.

In this case study where the data are limited, both Bayesian and Classical GLM were used to interpolate from a small number of scenarios. It can be difficult to ‘compare’ the results between Bayesian and classical GLMs [77, p. 6], although in this case study there were differences in the results, as shown in Table 2 and Figure 3. In this situation, we may compare the different implications of using Bayesian GLM versus Classical inference in terms of issues such as sample size, separation and stability, identifiability, exchangeability, and prediction.

1. This project encounters sample size issues, since only a few scenarios were elicited (5–18). In general Bayesian methods are often more useful than the classical approach in small sample situations [77,88]. In our situation, in particular, the Bayesian approach can estimate all two-way interactions albeit with wide posterior intervals, even with a few scenarios, and can show whether anything has been learnt about the interaction from the assessments provided. In contrast, classical GLM needs an additional scenario to be elicited for each additional interaction to be estimated. In addition, the Bayesian method applied here provides richer information on uncertainty about the contributions of each parent node ($p(\theta )$ versus $\hat{\theta }$ and SE(θ)).

2. When there are a large number of binary factors, then separation can arise, especially in Classical GLM [34, p. 1373]. With a small number of scenarios, we did not find any evidence in this case study of separation nor instability of estimates. The effect sizes $\hat{\beta }$ did not appear too large [42]. Both models produced stable estimates (e.g. see Figure 3).

3. When the coefficients of effect size estimates are not estimable or finite, this indicates that the corresponding regression model is not identifiable. In our case, we found that in both Bayesian and Classical GLMs, the main effects models were identifiable. In contrast, almost all interactions were estimable only under Bayesian GLM, albeit with wide uncertainty.

4. As referred to in Equation (12), the scenarios $s\in \{1,\dots ,S\}$ have been assumed to be conditionally exchangeable, although each sth scenario may be elicited with some carryover effects from previous scenarios $s-1,s-2,\dots$. Therefore, the order of eliciting the scenarios does not change the elicited outcomes. Exchangeability here is an advantage since we do not have to claim, in the Bayesian setting, the stronger assumption of elicited scenarios being independent, as required of a Frequentist analysis [33, Ch. 5].

5. By construction of the Bayesian method, stronger prior information can shift parameter estimates and predictions further from the data, so that the use of predictive performance measures may not be useful for evaluating or comparing models, with different kinds or levels of prior information. As mentioned by Gelman [33, Ch. 7, p. 167], with analogous notation provided in square brackets

   We are not saying that the prior cannot be used in assessing a model's fit to data; rather we say that the prior density is not relevant in computing predictive accuracy. Predictive accuracy is not the only concern when evaluating a model, and even within the bailiwick of predictive accuracy, the prior is relevant in that it affects inferences about θ and thus affects any calculations involving $p(y|\theta )$ [here y is equivalent to A and θ is equivalent to β]. In a sparse-data setting, a poor choice of prior distribution can lead to weak inferences and poor predictions.

In particular, small sample size (item 1) and identifiability (item 3) can affect the estimation of the interactions.

Another innovative contribution was to ‘fill out’ a large CPT, where parent nodes had more than three levels (see top Figure 9). This CPT was large in that it exceeded the usual size of CPTs. It would be possible to consider even larger CPTs, but that would require a larger set of scenarios to be elicited, which was beyond the scope of this paper. In contrast, standard software like CPT Calculator only allows up to three levels for each parent node. Even when we reduce the number of levels (from five to three) for the habitat suitability parent (bottom Figure 9), Bayesian GLM estimated a wider range of values, which was more ecologically sensible than for CPT Calculator. This could be because CPT Calculator uses a local form of regression that uses limited information to estimate each missing scenario, and moreover places undue emphasis on the worst scenario, which would dampen all the estimated values.

We may compare the different implications of encoding via CPT Calculator versus Bayesian Regression for Scenario-based Encoding (BRSE):

1. BRSE is global in that it uses all the elicited scenarios whereas CPT Calculator is local in that it uses only four scenarios to encode each missing CPT entry.

2. BRSE relaxes CPT Calculator's stringent assumption of stationarity when comparing two levels of any factor.

3. BRSE may flexibly adjust for expert assessments of interactions among factors, and in particular, the Bayesian approach ensures that all interactions are estimable, and reflect a non-informative prior when little information is elicited from experts on those interactions.

4. CPT Calculator relies heavily on accurate estimation of the worst scenario, which is included in every single interpolation factor and every estimate of missing CPT entries. However, BRSE adjusts the contribution of the worst scenario via average effects of every factor, which reduces the reliance on the worst scenario.

5. BRSEs make comparisons between probabilities on a more appropriate log odds scale rather than the naive linear comparisons adopted by CPT Calculator, which will be more problematic towards the extremes of zero or one.

Hence, we can conclude that Bayesian GLM is more useful than CPT Calculator for practical applications where it is important to conduct sensitivity or uncertainty analysis. In addition, Bayesian GLM is well-known to handle much larger (elicited) sets of scenarios (e.g. tens or hundreds of thousands in arm [35] and brms [17] packages), but here we were constrained by the application.

In some situations, experts may find it difficult to conceptualise a categorical variable with more than five levels. However, in others experts may consider a multi-category variable in order to estimate the CPT via Bayesian GLM. They would need to elicit each level of every factor at least twice, ignoring the level of other factors. This becomes complex and is beyond the scope of this paper to consider this experimental design problem. We have begun to explore this issue in future work. This paper aimed to simply present a new approach to designing elicitation and encoding of CPTs, which releases previous constraints that these tables be highly simplified. In future work, we will consider new designs, such as Fractional factorial designs [12], for choosing a few scenarios to ensure adequate coverage of even larger CPTs.

In summary, for large CPTs, our contribution is to adopt a fully Bayesian approach to obtain predictions of the remaining cell entries with uncertainty, which contrasts with existing literature in risk assessment [5] and ecology [18,30], which adopts a deterministic or confidence-interval interpretation of the uncertainty interval. Bayesian GLM is well suited to small samples [33, Ch. 16]; [79], and can provide the modeller with richer information, not only on which effects are most precisely estimated, and which have larger influence on the outcome, but also on whether sufficient information was obtained to estimate interactions. The latter is unlikely given that experts typically are time-poor, but may be possible that in some situations. As with design of data collection in any context, we may design elicitation to target what we believe to be the greatest sources of uncertainty or variability. Thus where experts are familiar and practised in giving accurate elicitations, it is possible to fully elicit a moderately large CPT. However, we suspect most often, investing more time in more accurately obtaining a few elicitations may be necessary to ensure what information is elicited is accurate [28]. In addition, more sophisticated experimental designs [12] such as randomized blocks [4] to streamline elicitation of multiple experts, not necessarily on the same scenarios but for different scenarios specified by groups for each expert. Then the workload of elicitation may be distributed among experts in gathering data.

Thus, as described above designing elicitation needs to consider how many scenarios are elicited from the full CPT, and how much effort is assigned to eliciting each scenario. In addition, it is important to carefully choose scenarios, as this can inadvertently introduce more biases. By construction, CPT calculator adopts a OFAT design for choosing scenarios, where the ‘control’ scenario is the ‘best’ one. This means that information elicited from experts is anchored around the best scenario, and hence interpolated values will be biased when considering scenarios where more than one factor falls towards the ‘middle’ or worst end of the spectrum. Thus, modellers should take care to ensure that sufficient scenarios are elicited in order to understand the effects of the identified interactions.

6. Conclusion

In summary, we have examined a new model-based approach to quantify and encode CPTs that define BNs, which moreover allows experts to specify their estimates with uncertainty. Since the objective was to minimize time spent during elicitation, only a minimal number of scenarios were elicited. A novel approach was taken to: (1) capture their best estimates together with their uncertainty in an intuitive way to represent the most plausible values of the probability (rather than the precision of their best estimate); and (2) use a statistical modelling approach (Bayesian GLM regression) to interpolate the remaining CPTs. In our case study, we found that Bayesian regression was more useful than classical GLM as it can explicitly describe wide uncertainty, especially about interactions, whilst being more precise for main effects. In this case study, with the minimal number of scenarios elicited according to a one-at-a-time design, the results demonstrated that the main effects models are better than interaction models because of the limited number of scenarios. However, if additional scenarios can be elicited then the Bayesian regression with interaction might prove to be better. In summary, our approach is suitable in practical applications particularly when CPTs are large (with more than three parents and/or the child node has more than three values) and in the presence of uncertainty, unlike CPT Calculator which uses a simple encoding model without uncertainty and cannot deal with large CPTs. Most importantly, we demonstrate that it is feasible to now spend less effort more wisely in quantifying CPTs in BNs. This can be achieved by eliciting more detailed information on fewer scenarios, and then encoding this information by harnessing more easily accessible statistical modelling, for Bayesian GLM.

Appendices.

Appendix 1. BN model nodes and their levels and definitions

Table A1. BN model nodes, their levels and definitions based on the model framework developed for feral pigs in northern Australia [30].

Node	Definition	Levels
Habitat Suitability	whether feral pig are able to breed and persist	Yes or No
Water quality	presence of accessible sources of potable water	Very good, Good, Moderate, Poor or Very poor
Presence of freshwater	presence of potable water	Yes or No
Water salinity	water may be absent or non-potable (salty/brackish)	Low, Moderate or High
Food	whether there are enough food resources available to meet the requirements of breeding	Very good, Good, Moderate, Poor or Very poor
Quality of food	presence and nutritional value of food resources	Low, Moderate or High
Duration	how long the food source is available (e.g. continual year-round, seasonal)	Low, Moderate or High
Accessibility	enable to access to food resources	Easy, Moderate or Hard
Seclusion	if there is enough disturbance refuge available to meet protection requirements of feral pigs	Very good, Good, Moderate, Poor or Very poor
Hunting	protection from hunting	High or Low
Human	disturbance stress from human interference	High, Moderate or Low
Cover	cover provided by understory vegetation	Good, Moderate or Poor
Shelter	if there is enough heat refuge available to meet protection requirements of feral pig	Very good, Good, Moderate, Poor or Very poor
Cover	shady vegetation cover	Good, Moderate or Poor
Hot climate	Heat stress conditions from daytime temperatures	Cool, Medium or Hot
Wallows	the place that feral pigs are relaxed in mud or water	Close, Medium or Far

Appendix 2. Wording the Outside-in elicitation

For the Outside-in approach, questions were structured like:

Let us think about 100 sites with the same scenario, where the presence of water is ‘yes’ and the salinity of water is ‘lo’. We want to know how many sites would have adequate quality of water. Of course you are 100% sure that between 0 and 100 sites are adequate. We need a lower and upper bound that are more useful than 0 and 100, and you can be less than 100% sure that the number falls between them. What is the smallest this number (of sites) can realistically be? What's the largest it could be? How sure are you that the number (of adequate sites) falls between them? What is your best estimate …?

Appendix 3. Extract of code used (Bayesian GLM)

This is an example of the model for main effects of shelter CPT using classical GLM [26] or Bayesian GLM [34] as referred to Section 3.2.5. The main difference is that Bayesian inference requires specification of priors, and uses a different library (arm).

Appendix 4. Food CPT

The Food CPT elicited information from experts that reflect their best estimates and uncertainties is attached in this appendix.

Table A2. Eliciting the expert's best estimate and uncertainties from experts for Food table.

	Quality	Duration	Accessibility	Lower	Best estimate	Upper
Best levels	High	High	Easy	0.80	0.90	0.99
Worst levels	Low	Low	Hard	0.05	0.08	0.15
3	Moderate	High	Easy	0.50	0.70	0.85
4	Low	High	Easy	0.10	0.23	0.53
5	High	Moderate	Easy	0.68	0.80	0.99
6	High	Low	Easy	0.08	0.58	0.70
7	High	High	Moderate	0.60	0.70	0.99
8	High	High	Hard	0.15	0.32	0.40

Appendix 5. Seclusion results

This appendix shows the Seclusion CPT elicited from experts that reflect their best estimates and uncertainties. Also, the figure for the difference between the predictions from Bayesian GLM and CPT Calculator of Seclusion CPT is attached in this appendix.

Table A3. Eliciting the expert's best estimate and uncertainties from experts for Seclusion table.

	Cover	Hunting	Humans	Lower	Best estimate	Upper
Best levels	Good	Low	Low	0.80	0.90	0.90
Worst levels	Poor	High	High	0.05	0.05	0.10
3	Moderate	Low	Low	0.50	0.65	0.70
4	Poor	Low	Low	0.20	0.35	0.40
5	Good	High	Low	0.70	0.80	0.90
6	Good	Low	Moderate	0.70	0.90	0.90
7	Good	Low	High	0.30	0.65	0.80

Figure A1.

Bayesian GLM $95\mathrm{\%}$ credible interval (left side) and Classical GLM $95\mathrm{\%}$ confidence interval (right side) of Food CPT for only model (red colour) versus main effects with all 2-way interactions (blue colour).

Figure A2.

For each scenario of Cover, Hunting and Humans of seclusion CPT: Comparison between predictions of CPT Calculator and predictions of Bayesian GLM, where the letters g, p, m, h and l symbolise to Good, Poor, Moderate, High and Low levels of scenarios respectively, e.g. ghh refers to good cover, high hunting and high humans. The plus signs refer to elicited values from experts and dots signs refer to the prediction that are not elicited. The Figure illustrated that there is slight differences between the predictions from Bayesian GLM and CPT Calculator.

Appendix 6. Water results

This appendix shows the Water CPT elicited from experts that reflect their best estimates and uncertainties. Also, the figure for the difference between the predictions from Bayesian GLM and CPT Calculator of Water CPT is attached in this appendix.

Table A4. Eliciting the expert's best estimate and uncertainties from experts for Water CPT.

Scenarios	Presence	Salinity	Lower	Best estimate	Upper
Best levels	Yes	Low	0.75	0.90	0.99
Worst levels	No	High	0.01	0.05	0.10
3	No	Low	0.01	0.05	0.10
4	Yes	Moderate	0.50	0.60	0.75
5	Yes	High	0.05	0.10	0.15

Figure A3.

For each scenario of water presence and water salinity: Comparison between predictions of CPT Calculator and predictions of Bayesian GLM, where the letters Y and N symbolise to Yes and No of water presence respectively while l, m and h symbolise to low, moderate and high levels of water salinity respectively, e.g. Yl refers to water presence, low salinity of water. The plus signs refer to elicited values from experts and dots signs refer to the prediction that are not elicited. The Figure illustrated that there is slight differences between the predictions from Bayesian GLM and CPT Calculator.

Figure A4.

Bayesian GLM $95\mathrm{\%}$ credible interval (left side) and Classical GLM $95\mathrm{\%}$ confidence interval (right side) of Water CPT for only model (red colour) versus main effects with all 2-way interactions (blue colour).

Figure A5.

Model fit criteria for Water CPT where y refers to the elicited data (dark density) and $y_{\mathrm{rep}}$ refers to the data simulated.

Also, Figure A3 shows the difference between the predictions from Bayesian GLM and CPT Calculator of Water CPT.

In addition, Figure A4 examines the comparison between Bayesian GLM $95\mathrm{\%}$ credible interval (left side) and Classical GLM $95\mathrm{\%}$ confidence interval of Water CPT.

Finally, Figure A5 shows the interpretation of model fit for Water CPT.

Appendix 7. Shelter results

This appendix shows the Shelter CPT elicited from experts that reflect their best estimates and uncertainties as shown in Tables 8 and 9.

Table A5. Eliciting the expert's best estimate and uncertainties from experts for Shelter CPT.

	Hot climate	Cover	Wallows	Lower	Best estimate	Upper
1	cool	good	close	1	1	1
2	cool	poor	far	1	1	1
3	cool	Moderate	close	1	1	1
4	cool	poor	close	1	1	1
5	cool	good	Medium	1	1	1
6	cool	good	far	1	1	1
7	medium	good	close	1	1	1
8	medium	poor	far	1	1	1
9	medium	Moderate	close	1	1	1
10	medium	poor	close	1	1	1
11	medium	good	Medium	1	1	1
12	medium	good	far	1	1	1
13	hot	good	close	0.95	0.99	0.99
14	hot	poor	far	0.01	0.01	0.05
15	hot	Moderate	close	0.40	0.80	0.80
16	hot	poor	close	0.30	0.50	0.70
17	hot	good	Medium	0.95	0.99	0.99
18	hot	good	far	0.01	0.10	0.40

Table A6. Eliciting the expert's best estimate and uncertainties from experts for Shelter CPT after ignoring climate factor.

Scenarios	Cover	Wallows	Elicited	Bayesian GLM	CPT Calculator
Best levels	good	close	0.95	0.99	0.99
Worst levels	poor	far	0.01	0.01	0.05
3	Moderate	close	0.40	0.80	0.80
4	poor	close	0.30	0.50	0.70
5	good	Medium	0.95	0.99	0.99
6	good	far	0.01	0.10	0.40

Figure A6.

For each scenario of shelter cover and wallows: Comparison between predictions of CPT Calculator and predictions of Bayesian GLM, where the letters g,m,p and N symbolise to good, moderate and poor of shelter cover, while c, m and f symbolise to close, medium and far levels of wallows respectively, e.g. pf refers to poor cover and far wallows. The plus signs refer to elicited values from experts and dots signs refer to the prediction that are not elicited. The Figure illustrated that there is slight differences between the predictions from Bayesian GLM and CPT Calculator.

Figure A7.

Bayesian GLM $95\mathrm{\%}$ credible interval (left side) and Classical GLM $95\mathrm{\%}$ confidence interval (right side) of Shelter CPT for only model (red colour) versus main effects with all 2-way interactions (blue colour).

Also, Figure A6 shows the difference between the predictions from Bayesian GLM and CPT Calculator of Shelter CPT.

In addition, Figure A7 examines the comparison between Bayesian GLM $95\mathrm{\%}$ credible interval (left side) and Classical GLM $95\mathrm{\%}$ confidence interval of Shelter CPT.

Finally, Figure A8 shows the interpretation of model fit for Shelter CPT.

Appendix 8. Suitability Habitat CPT

The Habitat suitability CPT elicited from three experts that reflect their best estimates and uncertainties are attached in this appendix.

Figure A8.

Model fit criteria for Shelter CPT where y refers to the elicited data (dark density) and $y_{\mathrm{rep}}$ refers to the data simulated.

Table A7. Eliciting the expert's best estimate and uncertainties from the first expert for Habitat Suitability CPT.

	Water availability	Food	Shelter	Seclusion	Lower	Best estimate	Upper
1	Very Good	Very Good	Very Good	Very Good	0.85	0.95	0.99
2	Very Poor	Very Poor	Very Poor	Very Poor	0.01	0.01	0.02
3	Good	Very Good	Very Good	Very Good	0.80	0.93	0.99
4	Moderate	Very Good	Very Good	Very Good	0.40	0.50	0.60
5	Poor	Very Good	Very Good	Very Good	0.11	0.25	0.30
6	Very Poor	Very Good	Very Good	Very Good	0.01	0.02	0.10
7	Very Good	Good	Very Good	Very Good	0.70	0.95	0.95
8	Very Good	Moderate	Very Good	Very Good	0.60	0.80	0.85
9	Very Good	poor	Very Good	Very Good	0.20	0.20	0.40
10	Very Good	Very Poor	Very Good	Very Good	0.05	0.10	0.32
11	Very Good	Very Good	Good	Very Good	0.90	0.95	0.99
12	Very Good	Very Good	Moderate	Very Good	0.90	0.95	0.99
13	Very Good	Very Good	Poor	Very Good	0.60	0.70	0.80
14	Very Good	Very Good	Very Poor	Very Good	0.20	0.20	0.60
15	Very Good	Very Good	Very Good	Good	0.70	0.90	0.90
16	Very Good	Very Good	Very Good	Moderate	0.70	0.90	0.90
17	Very Good	Very Good	Very Good	Poor	0.50	0.65	0.70
18	Very Good	Very Good	Very Good	Very Poor	0.01	0.10	0.20

Table A8. Eliciting the expert's estimate and uncertainties from the second expert for Habitat Suitability CPT.

	Water availability	Food	Shelter	Seclusion	Lower	Best estimate	Upper
1	Very Good	Very Good	Very Good	Very Good	0.99	0.99	0.99
2	Very Poor	Very Poor	Very Poor	Very Poor	0.01	0.01	0.01
3	Good	Very Good	Very Good	Very Good	0.99	0.99	0.99
4	Moderate	Very Good	Very Good	Very Good	0.55	0.55	0.55
5	Poor	Very Good	Very Good	Very Good	0.10	0.10	0.10
6	Very Poor	Very Good	Very Good	Very Good	0.01	0.01	0.01
7	Very Good	Good	Very Good	Very Good	0.70	0.70	0.90
8	Very Good	Moderate	Very Good	Very Good	0.50	0.50	0.50
9	Very Good	poor	Very Good	Very Good	0.30	0.30	0.30
10	Very Good	Very Poor	Very Good	Very Good	0.10	0.10	0.10
11	Very Good	Very Good	Good	Very Good	0.99	0.99	0.99
12	Very Good	Very Good	Moderate	Very Good	0.80	0.80	0.80
13	Very Good	Very Good	Poor	Very Good	0.01	0.01	0.01
14	Very Good	Very Good	Very Poor	Very Good	0.01	0.01	0.01
15	Very Good	Very Good	Very Good	Good	0.90	0.99	0.99
16	Very Good	Very Good	Very Good	Moderate	0.70	0.70	0.70
17	Very Good	Very Good	Very Good	Poor	0.20	0.20	0.50
18	Very Good	Very Good	Very Good	Very Poor	0.01	0.01	0.10

Table A9. Eliciting the expert's estimate and uncertainties from the third expert for Habitat Suitability CPT.

	Water availability	Food	Shelter	Seclusion	Lower	Best estimate	Upper
1	Very Good	Very Good	Very Good	Very Good	0.90	0.95	0.99
2	Very Poor	Very Poor	Very Poor	Very Poor	0.01	0.01	0.02
3	Good	Very Good	Very Good	Very Good	0.70	0.95	0.99
4	Moderate	Very Good	Very Good	Very Good	0.60	0.80	0.60
5	Poor	Very Good	Very Good	Very Good	0.01	0.10	0.18
6	Very Poor	Very Good	Very Good	Very Good	0.01	0.03	0.05
7	Very Good	Good	Very Good	Very Good	0.80	0.92	0.95
8	Very Good	Moderate	Very Good	Very Good	0.40	0.63	0.75
9	Very Good	poor	Very Good	Very Good	0.10	0.25	0.45
10	Very Good	Very Poor	Very Good	Very Good	0.01	0.03	0.11
11	Very Good	Very Good	Good	Very Good	0.80	0.90	0.99
12	Very Good	Very Good	Moderate	Very Good	0.80	0.85	0.90
13	Very Good	Very Good	Poor	Very Good	0.20	0.60	0.80
14	Very Good	Very Good	Very Poor	Very Good	0.01	0.10	0.60
15	Very Good	Very Good	Very Good	Good	0.80	0.95	0.99
16	Very Good	Very Good	Very Good	Moderate	0.70	0.90	0.99
17	Very Good	Very Good	Very Good	Poor	0.45	0.60	0.70
18	Very Good	Very Good	Very Good	Very Poor	0.08	0.10	0.20

Disclosure statement

No potential conflict of interest was reported by the authors.