Neural Bayes estimation and selection of complex bivariate extremal dependence models

Abstract

Likelihood-free approaches are appealing for performing inference on complex dependence models, either because it is not possible to formulate a likelihood function, or its evaluation is very computationally costly. This is the case for several models available in the multivariate extremes literature, particularly for the most flexible tail models, including those that interpolate between the two key dependence classes of ‘asymptotic dependence’ and ‘asymptotic independence’. We focus on approaches that leverage neural networks to approximate Bayes estimators. In particular, we explore the properties of neural Bayes estimators for parameter inference for several flexible but computationally expensive models to fit, with a view to aiding their routine implementation. Owing to the absence of likelihood evaluation in the inference procedure, classical information criteria such as the Bayesian information criterion cannot be used to select the most appropriate model. Instead, we propose using neural networks as neural Bayes classifiers for model selection. Our goal is to provide a toolbox for simple, fast fitting and comparison of complex extreme-value dependence models, where the best model is selected for a given data set and its parameters subsequently estimated using neural Bayes estimation. We apply our classifiers and estimators to analyse the pairwise extremal behaviour of changes in horizontal geomagnetic field fluctuations at three different locations.

Supplementary Information

The online version contains supplementary material available at 10.1007/s10687-025-00521-8.

Introduction

Motivation

Recent developments in bivariate extreme value modelling have produced new classes of models that allow for interpolation between the two key tail dependence regimes of asymptotic dependence and asymptotic independence; see for instance (Engelke et al. 2019). These models simplify the approach to bivariate extremal modelling, by eliminating the need to pre-determine a dependence regime using unreliable empirical diagnostics. However, evaluating their likelihood functions often relies on numerical integration and inversion of functions, as well as censoring of non-extreme values, which makes likelihood evaluation burdensome. In other situations, the likelihood function might not be available at all. Nonetheless, despite the likelihood function being intractable or unavailable, it is often possible to simulate data from the model; this allows for the use of simulation-based likelihood-free algorithms to estimate model parameters.

Traditional likelihood-free inference approaches

One simulation-based approach is the pseudo-marginal Markov chain Monte Carlo (MCMC) sampler proposed first by Beaumont (2003) and later formalised by Andrieu and Roberts (2009). When the target distribution is intractable, usually due to the likelihood function, pseudo-marginal MCMC is able to approximate the target function using an unbiased estimator obtained through importance sampling. The expected value of such an estimator corresponds to the true target distribution, enabling the algorithm to correctly sample from it. Pseudo-marginal MCMC methods, however, perform poorly when dealing with high-dimensional parameter vectors (Alenlöv et al. 2021) or when the distribution of the proposed estimator is, for example, heavy tailed, resulting in difficulties converging to the target distribution (Murray and Graham 2016). Another commonly used likelihood-free procedure is approximate Bayesian computation (ABC; see, e.g., Lintusaari et al. 2017 and Sisson et al. 2018). Specifically, this can be seen as a rejection sampling algorithm, where the model parameters are generated from a prior distribution and subsequently accepted or not based on the distance between a data sample simulated from the model, given the simulated parameters, and the original data sample, often evaluated based on informative summary statistics. Choosing a suitable prior distribution, and defining how similar the data samples are, constitute major drawbacks of using ABC to perform inference. An inadequate prior distribution might lead to misleading posterior estimates, particularly in situations where the selected summary statistics are not very informative. Conversely, a prior distribution which is too informative can result in a posterior that is skewed or biased towards the prior distribution, even if it leads to less variable estimates. Wood (2010) propose the ‘synthetic likelihood’ method, which constructs an approximate likelihood function by assuming that user-defined summary statistics follow a multivariate normal distribution. This approach is usually easier to tune than ABC and computationally more efficient, especially with higher dimensional data sets (Price et al. 2018), but its usual underlying Gaussian assumption makes it inflexible in some cases, which may lead to sub-optimal inferences.

Neural bayes inference for extremes

More recently, there has been a growing interest in likelihood-free estimation methods which use neural networks; see Zammit-Mangion et al. (2025) for an in-depth review. The extremes literature has also started to be impacted by this new inference paradigm, mostly in the spatial setting; see for instance (Dell’Oro and Gaetan 2025; Lenzi et al. 2023; Majumder and Reich 2023; Majumder et al. 2024; Rai et al. 2024; Richards et al. 2024b, a; Sainsbury-Dale et al. 2024, 2025) and Walchessen et al. (2024). One approach for leveraging neural networks for likelihood-free inference is to obtain a point estimate of the vector of model parameters through a neural Bayes estimator (Sainsbury-Dale et al. 2024). Training the neural network to build such an estimator can be computationally expensive; however, this step only needs to be done once, with estimates subsequently obtained in milliseconds with new data, using a single graphics processing unit (GPU). This means that neural Bayes estimators are amortised (Zammit-Mangion et al. 2025), which allows for their repeated use at almost no extra computational cost (see, e.g. Richards et al. 2024b for a compelling data illustration). Therefore, neural Bayes estimators are an appealing, and much faster, avenue to performing inference compared to state-of-the-art likelihood-based methods. Moreover, unlike other likelihood-free methods, such as ABC, this approach automatically learns the relevant summary statistics for the inference problem at hand. Given the computational complexity of the models of interest, this is the approach we take in this paper. Whilst the methodology developed for neural Bayes estimation has been mostly applied in the spatial and temporal contexts, we are interested in exploring its applicability in a simple bivariate setting, while allowing for censored data inputs. In order to achieve this, an appropriate neural network architecture will need to be designed, along with suitable prior choices. While the number of parameters to estimate may be similar to typical spatial models, the inference procedure may be more challenging in a bivariate setting compared to a spatial context, since there are fewer distinctive features (e.g., location, distance) in the data that the neural Bayes estimator can learn from.

When neural Bayes estimators are adopted for inference, typical model selection techniques, such as the Akaike information criterion (AIC) or the Bayesian information criterion (BIC), are often not available as they require knowledge of the likelihood function; see, e.g., Claeskens and Hjort (2008) for details on model selection criteria. Therefore, having a likelihood-free way of selecting the best model, for a given set of candidate models fitted to a certain data set, is desirable. Radev et al. (2023) show how the marginal likelihood (and hence, Bayes factors useful for model selection) may be approximated using a neural network-based approach. Similarly, neural methods targeting the likelihood-to-evidence ratio (see, e.g., Cranmer et al. 2016; Hermans et al. 2020) could be used for model selection using likelihood ratios. These methodologies, however, rely on full posterior and/or full likelihood neural approximations. Ahmed et al. (2022) and Wixson and Cooley (2024), on the other hand, propose using neural networks as classification tools for testing the extremal dependence type of a data set, however not in the context of model selection. Whilst the former only focus on spatial processes, the latter apply the classifier to bivariate data sets. More recently, Ahmed et al. (2024) propose using neural networks for selecting spatial max-stable processes. To do so, the extremal concurrence probability (Dombry et al. 2018) is used as a summary statistic to extract the spatial features of a data set. However, if such summary statistic is not sufficient, its use as input to the neural network might result in a weaker ability to distinguish between key features. Here, we make a step further and instead propose a model selection procedure based on a classification problem, whereby a neural network architecture (analogous to that used for parameter estimation) automatically learns relevant summary statistics for the model selection problem at hand. More specifically, a neural Bayes classifier, designed to learn the distinguishing features of each model, is used (once trained) to efficiently estimate the posterior probability of a data set arising from a certain model. The obtained posterior probabilities can then be used as weights in Bayesian model averaging (e.g., Hoeting et al. 1999) type procedures, if desired.

Beyond parameter estimation, model selection and hypothesis testing, work at the interface between statistics of extremes and deep learning is not new and has been of growing interest recently; see, e.g., Cannon (2010); Cisneros et al. (2024); Pasche and Engelke (2024); Richards and Huser (2022) for univariate extreme regression modelling, and Murphy-Barltrop et al. (2024) for extremal dependence modelling.

Main contributions and paper outline

In this paper, we contribute to the neural inference literature on extremes with the aim to provide a supervised learning toolbox for simple fitting and selection of complex bivariate extremal dependence models, which avoids the subjective, and often awkward, selection of summary statistics. We start by exploring the utility of neural Bayes estimators in this specific setting; this is done through assessment of the estimation accuracy of the model parameters and key dependence measures. The uncertainty of the neural Bayes estimators is assessed and compared through two different approaches: a non-parametric bootstrap approach, and an estimator trained to target posterior quantiles. We then outline the neural Bayes classifier, and examine its success for model selection. When available, we compare its performance to a likelihood-based information criterion. The end goal is to make the entire statistical pipeline amortised. First, the best model for a given data set is selected through the neural Bayes classifier, and then estimates of the model parameters are obtained through a neural Bayes estimator.

This paper is organised as follows: in Section 2, we introduce the methodology used for parameter estimation using neural networks for both uncensored and censored data, and describe our model selection procedure based on a classification task. Section 3 presents an overview of bivariate extreme value modelling and introduces the models of interest for which likelihood-based inference is burdensome. Simulation studies assessing our proposed inference and model selection frameworks are discussed in Section 4. We then apply the proposed toolbox to study the pairwise extremal behaviour of the changes in horizontal geomagnetic field fluctuations between three locations in Section 5, followed by a conclusion in Section 6.

Inference methodology

In this section, we review background on neural point estimation using the methodology developed by Sainsbury-Dale et al. (2024) and Richards et al. (2024b) and describe our neural approach to model selection. In Section 2.1, we review neural Bayes estimators, firstly outlining the general framework when the sample size is fixed and known, and data are fully observed (i.e., uncensored). We then explain how to adapt the estimation procedure to account for variable sample size and censored data, respectively. In Section 2.2, we present our neural Bayes classifier for model selection and describe how to construct it. Implementation details are given in Section 2.3. Although we later focus on bivariate models, we use data dimension d in this section for generality.

Neural bayes estimators

General framework

Let be n independent and identically distributed random vectors, and let their collection be represented by We assume that admits the parametric joint density where is the vector of parameters. A point estimator maps data to parameter estimates from the parameter space i.e., Given a non-negative loss function a Bayes estimator minimises a weighted average of the risk at which may be expressed as

2.1

where is a prior measure for Equation (2.1) is known as the Bayes risk. It is key to realise that Bayes estimators are functionals of the posterior distribution induced by the prior . In particular, if is the squared error loss, is the posterior mean of . If is instead the absolute error loss, is the posterior median, and alternative posterior quantiles can be estimated using the quantile loss function; see Section 4.2 for further details. Under suitable regularity conditions and the squared error loss, these estimators are consistent and asymptotically efficient; see for instance (Lehmann and Casella, 1998, Ch. 5 and 6).

In practice, however, Bayes estimators are rarely available in closed form, and the Bayes risk in Equation 2.1 is difficult to evaluate. This can be overcome by approximating these estimators using a neural network, since these are universal function approximators (Hornik et al. 1989). In this context, a neural point estimator is constructed as a neural network, with parameters that returns a point estimate from data input Bayes estimators may thus be approximated with where and

2.2

In Equation 2.2, is a training set of size consisting of draws from the the prior and corresponding data set realisations sampled from the model given , i.e., from The neural point estimator is called a neural Bayes estimator (NBE) as it minimises a Monte Carlo approximation (2.2) of the Bayes risk (2.1); see Sainsbury-Dale et al. (2024) for more details.

The discrepancy between the neural Bayes estimator and the true Bayes estimator will depend on a few factors, one of which is the neural network architecture. Through a judicious choice of architecture, NBEs can be enforced to satisfy the fundamental property that for any vector whose elements are permutations of the independent replicates in this can be achieved by exploiting a neural network architecture known as DeepSets (Zaheer et al. 2017; Sainsbury-Dale et al. 2024). Let and be two multilayer neural networks parameterised by and respectively. Following (Sainsbury-Dale et al. 2024), the NBE is then represented as

2.3

where the average is computed componentwise, denotes a vector of learnt summary statistics, and are the parameters of the neural networks and . In the multivariate unstructured setting, a multilayer perceptron (MLP) neural network may be used for both and see Section 2.3 for further details. A schematic of the DeepSets architecture is shown in Section S1 of the Supplementary Material.

More recently, Rödder et al. (2025) have provided theoretical and asymptotic guarantees for NBEs. In particular, the authors show that, under the squared error loss function, an MLP with ReLU activation function (see Section 2.3), and other regularity assumptions, these estimators are fully efficient compared to the corresponding analytical Bayes estimators, and are consistent in a Bayesian sense. They do so by decomposing the empirical risk (2.2) into the analytical Bayes risk, an approximation error term, a generalisation error term, and an algorithm error term, which all converge to zero under an appropriate asymptotic regime.

The performance of the NBE is inherently influenced by the choice of the prior distributions. While an informative prior might reduce the volume of the parameter space, and allow lower values for in Equation 2.2, this restricts the use of the trained estimators across several applications; when the latter is desirable, a vague prior is advisable (Sainsbury-Dale et al. 2024). When the likelihood is available, and feasible, informative prior distributions can be constructed using likelihood-based estimates (Lenzi et al. 2023). Alternatively, informative priors can be obtained by sequentially incorporating the knowledge from previously estimated posterior distributions; however, this is no longer an amortised approach, as the neural network must be retrained whenever the prior changes.

Variable sample size

When training a neural Bayes estimator on a data set with a fixed number, n, of replicates, this estimator will generally not be Bayes for data sets with a different sample size Therefore, in order to ensure that the trained NBE approximately minimises the Bayes risk for varying sample sizes n, Sainsbury-Dale et al. (2024) propose two approaches: either obtaining a piecewise neural Bayes estimator by pre-training the estimator for specific fixed sample sizes (Goodfellow et al. 2016), or treating the sample size as a random variable N; we adopt the latter in our work.

Let us assume that the sample size N follows a discrete uniform distribution, that is where for and Further, the sample size N is assumed independent of the model parameters This extra random variable modifies the Bayes risk function (2.1), which can now be approximated by

where is a training set of size consisting of sample sizes n drawn from the distribution, parameters drawn from the prior and corresponding data set realisations drawn from the model given n and . Thus, during training, it is now necessary to simulate the sample size along with model parameters from the prior and replicated data from the model; however, the general inference method remains the same.

Censored data

The use of neural Bayes estimators for censored data was later considered by Richards et al. (2024b). This is essential in multivariate models aimed at capturing the extremal dependence structure and common practice in multivariate extremes; see Ledford and Tawn (1996) and Smith et al. (1997) for early examples of this approach. In these models, low observations are often censored to prevent these non-extreme values from affecting the estimation of the tail dependence. As shown by Huser et al. (2016), treating the censored data as observed leads to more biased estimators.

Consider the random vector and let be the inverse cumulative distribution function (cdf) of variable Various censoring schemes can be adopted. One possibility, used by Richards et al. (2024b), is to censor the observations that fall below a high marginal -quantile, i.e., if then is treated as censored. Instead, in this paper, we censor the observations only if all the components are below their respective -marginal quantile, i.e., if then the entire vector is treated as fully censored, otherwise, if at least one component of has a value above its marginal -quantile, the entire vector is treated as uncensored. We do so for consistency in later comparisons between models (see Sections 3.3 and 4.3), as this is the only censoring scheme that is implemented for likelihood-based inference in some of the models we consider. In addition, this censoring scheme can be less computationally demanding for likelihood-based estimation.

In order for the neural Bayes estimator to account for censored data, Richards et al. (2024b) propose standardising data to a common marginal scale and setting the censored observations to some constant , yielding To improve performance of the NBE, this constant c is set to a value outside of the support of the data. In order for information on the censored observations to be passed to the NBE, Richards et al. (2024b) propose additionally creating a one-hot encoded vector that identifies which indices of are censored (taking the value ), and which are not (taking the value ). Then, the neural Bayes estimator is trained using an augmented data set containing the data and the indicator vector that is Passing as the input to the neural network in place of in Equation 2.3 is sufficient to ensure all information about the censoring scheme is given to the NBE. In contrast to the approach of Richards et al. (2024b), under the adopted censoring scheme, each vector is either a d-dimensional vector of zeros or a d-dimensional vector of ones. Nevertheless, we believe that incorporating as an input to the neural network remains beneficial, as it carries information about the number of exceedances during training. Moreover, we propose handling the augmented data set with a bilinear layer (see Section 2.3) as the input layer of the neural network instead.

Similarly to the sample size, these NBEs are only (approximately) optimal when applied to data sets where the censoring level is kept the same as the one used for training. However, having a single neural estimator which performs well for any valid —or a subset thereof—is often desirable. This can be achieved by feeding as an extra input to the outer neural network as

where as in (2.3) with in place of .

In this context, the censoring level must be treated as a random variable T, which requires an additional prior. As we are not interested in censoring too low, a prior concentrated around high quantiles (i.e., with probability values close to 1) is a natural choice. Thus, each data set now has a censoring level associated with it, making it possible to have different censoring levels for the different training parameter vectors and corresponding data samples. The Monte Carlo approximation of the Bayes risk in this case takes now the form

where is a training set of size consisting of censoring levels drawn from the continuous distribution with density over sample sizes n drawn from the discrete distribution as above, parameters drawn from the prior and corresponding data set realisations , drawn from the model given n and that are masked at the simulated censoring level and augmented with the threshold exceedances indicator vector as described above.

Model selection with neural bayes classifiers

From a Bayesian perspective, model selection is naturally performed by treating the model type as a random variable, M, which is inferred along with model parameters, based on data set . Consider a finite collection of candidate parametric models, such that M takes values in , and assume that a priori with . In our experiments, we specify a uniform prior, i.e., for all , indicating that the data have an equal probability of being drawn from each model a priori; however, other choices are possible. Our interest thus lies in approximating the posterior distribution of , i.e., the distribution of —or functionals thereof—where is the combined vector of model parameters for all candidate models, and is the parameter vector for the m-th model. This joint inference task may be drastically simplified (without loss) by noting that the conditional distribution of can be written as the product between the distribution of and that of , both of which can be inferred separately. Inferring can be further simplified by splitting the problem into model-specific inference tasks, i.e., by separately inferring for each . This can be achieved efficiently by training different NBEs for each specific candidate model ; recall Section 2.1. Therefore, to perform amortised model selection, the only missing piece is a fast inference technique to compute, or rather approximate, the distribution of . Bayes’ theorem implies that

2.4

where is the parameter space of is a prior measure for and denotes the density of (evaluated at the value ) given and In other words, is the parametric joint density of the m-th candidate model (for n independent replicates) for fixed parameter vector Equation (2.4) thus shows that the distribution of is unavailable when the individual model densities are intractable.

Nevertheless, similarly to amortised parameter estimation, it is possible to construct a neural network that approximates the true discrete distribution of for any data input . Let be the true (target) posterior probabilities of , and let denote the corresponding approximate posterior probabilities (appropriately constrained to be positive and sum to one), obtained as the output of a neural network parameterised by the vector . We further write , and for the densities of (marginalised over both and M), (marginalised over only), and , respectively, and let and be the sample spaces of and , respectively. Following the arguments in Zammit-Mangion et al. (2025), the required inference task reduces to finding the optimal that minimises the expected forward Kullback–Leibler divergence between and , that is

2.5

where is a training set of size consisting of models m drawn from the prior for M with probabilities , model parameters drawn from the prior (i.e., given the model m), and corresponding data set realisations drawn from the fixed model m given parameters . Note that the fourth equality above is justified by a simple application of Bayes’ theorem. With data , amortised model selection can thus proceed based on by selecting the model that maximises the inferred posterior probability This is akin to a classification problem, where each model type is treated as a different ‘class’ (among K classes), and where the data are labelled according to the class that is the most likely a posteriori. The optimisation in Equation 2.5 is indeed identical to minimising the categorical cross-entropy loss function, which is a popular and natural choice in multiclass classification problems. Thus, by analogy with NBEs, we call our amortised model selection tool the neural Bayes classifier (NBC). If suitably trained, the NBC is expected to perform well for any data drawn from one of the K models, provided that is sufficiently large in Equation 2.5. However, as with NBEs, the architecture of NBCs should depend on the structure of the data at hand and the desired level of amortisation. In particular, with replicated data (i.e., with ), the DeepSets architecture introduced in Section 2.1 can be applied, and similar techniques to those used for parameter estimation can be employed to ensure the NBC remains suitable for censored data, variable sample sizes, and/or variable censoring levels. We note, however, that there is no restriction in using a different neural network architecture to that used for the NBEs.

Implementation details

In this work, a multilayer perceptron (MLP) neural network is used for both and in the DeepSets construction (2.3), for both parameter and model selection procedures. Two key components of a standard MLP are its activation functions and vector of parameters , with the latter containing what is often referred to as weights and biases. The activation functions introduce non-linearity into the MLP construction, allowing the network to learn and represent the complexities of the underlying data. The parameters, on the other hand, determine the strength (weights) of the connection between two neurons (i.e., nodes of the neural network), and shift the input of the activation functions (biases). Consider a generic MLP with L layers, and let and represent the matrix of weights and vector of biases (of appropriate dimensions) of layer respectively. The MLP is constructed by taking the input and transforming it into the output as

where are often denoted as the hidden states (Zammit-Mangion et al. 2025), and is the activation function of the l-th layer (). As mentioned in Section 2.1.3, a bilinear layer is used as the input layer of the neural network when handling censored data. As opposed to the layer defined above, this type of layer allows for two separate inputs, that may be of different dimension, to be combined into a single input. In particular, given the augmented data set for the k-th element of the first hidden state for each data replicate i takes the form with , for where is the output dimension of This corresponds to a sum of elementwise interactions between the standardised data input and the indicator vector As before, the n data replicates are combined using the DeepSets framework (recall Sections 2.1.1 and 2.1.3). We note, however, that with the adopted censoring scheme (i.e., with either being a d-dimensional vector of zeros or ones), identical results are obtained when considering the original data input instead.

Unless stated otherwise, the Rectified Linear Unit (ReLU) activation function is used when training the neural Bayes estimator and classifier, except in the final layer L. Writing for (and for ), the ReLU function returns the same value if the corresponding element of is positive, and 0 otherwise, i.e, applied elementwise. For the final layer L of the network (recall the DeepSets framework in (2.3)), different activation functions are used so that constraints on the support of the parameters are satisfied. In particular, when training the NBE, if the j-th parameter is strictly positive, the softplus activation function is applied, i.e., where denotes the j-th element of the vector . If the parameter is bounded in the interval [a, b], a layer compression is used instead; such a layer uses a logistic function restricting the j-th parameter to be within [a, b]. The identity function, is used for parameters whose support is the real line. When training the NBC, the softmax activation function is used in the final layer L so that the output is a valid vector of probabilities, i.e., all elements of are positive and . More specifically, writing , the softmax activation function takes the form with componentwise operations again.

As mentioned in Sections 2.1 and 2.2, neural Bayes estimators and classifiers are trained by minimising a specific objective function defined either as the empirical Bayes risk (for parameter estimation) or as the expected Kullback–Leibler divergence (for model selection), with respect to the neural network parameters More specifically, the training process involves learning the optimal parameters that map data inputs to parameter estimates or posterior probabilities. This optimisation is done via back-propagation and the stochastic gradient descent (SGD) algorithm, where parameters are iteratively updated to minimise the selected objective function. Moreover, during training, two types of data sets are used: the training and validation sets, denoted respectively by and from now on. Both data sets are passed through the network and are refreshed after every epoch; here an epoch is defined as one full cycle through all the input data in the training set during the SGD process. Precisely, the training set contains data used to train the model, which are used to update the parameters of the network by minimising the objective function using SGD. On the other hand, the validation set does not contribute to the parameter updates; instead, it is used to assess the ability of the model to generalise to new data by evaluating the same objective function on the validation set, which helps to avoid overfitting, and it is used to define an early-stopping criterion for the algorithm (see, e.g., Prechelt 1998). Finally, the performance of the trained estimator to model new data is objectively assessed by evaluating the same objective function on a test set, which is not used during training (nor validation) at all. All the computations are performed using the NeuralEstimators (Sainsbury-Dale 2024) and Flux (Innes 2018) packages in Julia; see https://msainsburydale.github.io/NeuralEstimators.jl/dev/ and https://fluxml.ai/Flux.jl/stable/, respectively, for the full documentation.

Bivariate models of interest

From now on, we work in the bivariate setting. Specifically, we focus on the modelling of the joint tail behaviour of the (single) random vector We note, however, that the inference methodology can be applied to higher dimensions. We are particularly interested in bivariate models that are suitable for the modelling of both types of extremal dependence structures. More specifically, we focus on flexible models that allow interpolation between asymptotic dependence and independence, as well as on the weighted copula model (WCM) proposed by André et al. (2024), which is designed to represent both the body and tail of a data set. We aim to provide a tool for fast inference for a variety of bivariate models exhibiting complex dependence structures. Evaluation of their likelihood functions relies heavily on numerical integration and inversion of functions; this results in computationally costly likelihood-based inference procedures and may otherwise limit the use of these models in practice. Furthermore, likelihood-based inference for the WCM, which is a dynamic mixture model, is currently infeasible when one of its components is taken as one of the models able to interpolate between the two classes of extremal dependence. Section 3.1 reviews basics of copula modelling, while some background on extremal dependence measures is given in Section 3.2. The models of interest are introduced in Sections 3.3 and 3.4.

Copula modelling

The dependence between variables and can be characterised by means of copulas. Let and be the margins and let denote their joint distribution function. According to Sklar’s theorem (Sklar 1959), the underlying copula of is obtained as

When and are continuous variables, the copula C is unique and represents the joint distribution function of where and are random variables. This result is useful since it is sufficient to marginally transform the data to a uniform scale through the probability integral transform and represent their joint behaviour via a copula model C. When it exists, the copula density can be obtained by taking the second derivative of C with respect to and as

where is the probability density function (pdf) of variable for and is the joint pdf of and When does not have an explicit form, it often needs to be computed through numerical integration; this might also be necessary for Additionally, inversion techniques are often required to compute All of these can require substantial computational resources.

Bivariate extremal dependence measures

When interest lies in the joint extremes of a bivariate random vector, a key element is to correctly identify its extremal dependence behaviour, i.e., whether large values in different components of this vector are likely to occur simultaneously or not. Misidentifying the extremal dependence structure may indeed lead to inaccurate representations of the extremes, and incorrect extrapolations (see, e.g., Ledford and Tawn 1996, 1997; Heffernan and Tawn 2004; Huser and Genton 2016). Intuitively speaking, asymptotic dependence (AD) is present if the most extreme values of the components of the random vector can occur together, and asymptotic independence (AI) is present otherwise. This extremal behaviour is often quantified through the tail dependence coefficient (see, e.g., Joe 1997) and/or through the residual tail dependence coefficient (Ledford and Tawn 1996). The coefficient can be obtained as when it exists, with

3.1

The vector is asymptotically independent if and asymptotically dependent if Given a function that is slowly-varying at zero (i.e, as for any ), Ledford and Tawn (1996) assume the joint tail may be expressed as

3.2

If and as then is asymptotically dependent with otherwise it is asymptotically independent, with larger values of indicating stronger dependence. Similarly to given in Equation 3.1, a sub-asymptotic version of can be obtained from Equation 3.2.

Taken together, provides a summary measure of dependence within the AD class (with ), while provides a summary within the AI class (with ). Available models in the multivariate extremes literature are often only suitable for one extremal dependence class. That is, aside from boundary points of the parameter space, traditional models either yield or but cannot span across both classes. However, in recent years, more flexible methods which are able to capture both types of dependence have been proposed; see for instance Wadsworth et al. (2017); Huser and Wadsworth (2019) and Engelke et al. (2019). Each of these can be written as a random scale construction, which we introduce in the following section.

Random scale construction models

A range of bivariate dependence models can be constructed using a random scale representation based on ‘radial’ and ‘angular’ coordinates. More specifically, a random scale mixture vector is constructed as follows

3.3

where is the ‘radial’ variable and is assumed to follow a non-degenerate distribution, and is the vector of ‘angular’ components. Different models can be obtained by varying the distributions and constructions of R and Depending on the precise specification, these may be able to interpolate between the two regimes of extremal dependence; see Engelke et al. (2019) for a detailed overview of the dependence properties arising from this construction. Interest lies in exploiting the copula C of these flexible models. Moreover, since these aim at capturing the extremal dependence of the vector non-extreme values are often censored to prevent their influence on the joint tail. We now present four particularly interesting bivariate models with construction (3.3), each of which can yield both and depending on their parameter vectors, with the transition between these two regimes occurring at interior points of the parameter space.

Model W

Let and let R follow a generalised Pareto distribution (GPD) with scale parameter 1 and shape parameter i.e, with cdf Wadsworth et al. (2017) propose a model with

Given the model construction, is assumed to hold only when is large. This model is able to smoothly interpolate between the two classes of asymptotic dependence through the parameter For is asymptotically dependent with

If then asymptotic independence is present with and More details can be found in Wadsworth et al. (2017).

Model HW

Huser and Wadsworth (2019) detail a model construction providing flexible extremal dependence structures for spatial processes. However, they also propose a bivariate model able to transition between different types of dependence. In this model, both R and the vector are marginally Pareto distributed with different shape parameters. More specifically, the marginal cdfs of each variable are given by and with defined analogously, for Additionally, we assume here that follows a bivariate Gaussian copula with correlation parameter see also Gong and Huser (2022a, 2022b).

This model is able to interpolate between the two classes of extremal dependence through the parameter when the tail of R is heavier than and so is asymptotically dependent with

When is asymptotically independent with

where is the residual tail dependence coefficient (3.2) of vector More details can be found in Huser and Wadsworth (2019).

The final two models were proposed by Engelke et al. (2019). Similarly to the model introduced by Wadsworth et al. (2017), let

Model E1

For the first model, R follows a Weibull distribution with distribution function and the angular components are constructed as follows

where and For this model, the extremal dependence is controlled by when and are asymptotically independent with and If they are asymptotically dependent with

Model E2

For the second model, with as in Wadsworth et al. (2017). The angular components and are now independent of each other and distributed as V, that is The extremal dependence of this model is determined by In particular, if

and are asymptotically dependent. When the variables are asymptotically independent with

Although Models E1 and E2 were proposed in Engelke et al. (2019), to the best of our knowledge they have not been used for model inference or in concrete data applications in the literature previously.

Weighted copula model

André et al. (2024) propose a model that is able to accurately represent both the body and tail regions of a data set, while ensuring a smooth transition between them. Let denote the density of a copula tailored to the tail, with parameter and denote the density of a copula tailored to the body, with parameter For , the density of the proposed model is given by dynamically mixing with as follows

3.4

where is a dynamic weighting function, is the vector of model parameters, and a normalising constant. Note that model (3.4) does not have uniform margins in general, which makes likelihood-based inference difficult. The weighting function is dynamic and specified such that it is increasing in and for a fixed value of in particular, more weight is given to for small values of and more weight is given to for large values of Similarly to the models introduced in Section 3.3, the interest lies in exploiting the copula of density (3.4), where and are the marginal cdfs of the cdf H corresponding to h in (3.4). As shown in André et al. (2024), the model shows some interesting features regarding extremal dependence, with potentially having an influence on depending on the weighting function chosen. More details can be found in André et al. (2024).

Simulation studies

Several simulation studies are performed to assess the trained neural Bayes estimators in different scenarios. We present selected studies in this section, and give the remaining ones in the Supplementary Material. All general settings and priors are outlined in Section 4.1. In Section 4.2 we then study the performance of NBEs in estimating the model parameters of the models described in Sections 3.3 and 3.4. We assess the efficacy of the neural Bayes classifier for model selection in Section 4.3, focusing on the four models from Section 3.3. In Section 4.4 we investigate the performance of the NBCs and NBEs trained for model selection and parameter estimation, respectively, in misspecified scenarios.

General settings

In all simulations, the sample size n and, when applicable, the censoring level are assumed random and independent realisations of variables distributed, respectively, as and Each parameter of the four models mentioned in Section 3.3 are also assumed independent a priori and uniformly distributed. In particular, for Model W (Wadsworth et al. 2017) we take for the parameter of the Beta distribution, and for the shape parameter of the GPD. For Model HW (Huser and Wadsworth 2019), we take and For Model E1 (Engelke et al. 2019), we take for the parameter of the Beta distribution, for the Weibull parameter, and For Model E2 (Engelke et al. 2019), we assume that and For the weighted copula model from Equation 3.4, we take the weighting function to be with as the prior for the weighting function parameter; based on previous analysis of this model, ensures that there is a good representation of both copula components, and in each drawn sample.

In order to train the NBE for parameter estimation, training and validation sets are generated (recall Sections 2.1 and 2.3). For the training set , we sample parameter vectors independently, and for each of them we sample five different data set realisations from the model, each associated with a different (randomly sampled) sample size n and censoring level (when applicable). Therefore, the training set for the NBE is of size For the validation set , we use the same sampling strategy, but we only sample parameter vectors, thus resulting in a set of size In order to train the NBC for model selection (recall Section 2.2), a similar approach is used, but the model is now also treated as random. For the training set , we start by generating models m, but only sample a single parameter vector, sample size, censoring level (when applicable), and corresponding data set for each model. Therefore, the size of the training set for the NBC is For the validation set, we use the same strategy but only generate models, resulting in a validation set of size In all simulations, the rank transform is used to transform the margins of to an approximate uniform scale, as required when fitting our copula models.

The number of layers assumed for each neural network and (recall (2.3)) and its parameters are determined experimentally, and in order to reduce the computational intensity, we adopt the ‘simulation-on-the-fly’ technique; see Sainsbury-Dale et al. (2024) for more details. We also adopt an early stopping rule whereby the training step finishes if the objective function (i.e., empirical Bayes risk for the NBE or expected Kullback–Leibler divergence for the NBC) computed based on the validation set has not decreased in 5 consecutive epochs for parameter estimation and 10 consecutive epochs for the model selection procedure. All the simulations are performed using a high-end computing cluster with a NVidia V100 32 GB GPU hardware with 192 GB of memory; see https://lancaster-hec.readthedocs.io/en/latest/gpu.html for more details (last accessed on 02/09/2025). For reproducibility and to make our methodology available to a broad readership, all the trained NBEs for parameter estimation and NBCs for model selection are available on https://github.com/lidiamandre/NBE_classifier_depmodels.

Parameter estimation assessment

We take the absolute error loss function in Equation 2.2 and its analogues, thus targeting the marginal posterior median. We first demonstrate the performance of NBEs for uncensored data from the weighted copula model from Section 3.4, and for censored data from one of the random scale mixture models mentioned in Section 3.3. The neural network architecture used for parameter estimation is given in Table S1 of the Supplementary Material. Finally, the assessment of the NBEs is done on a test parameter set with 1000 parameter vectors and corresponding test data realisations, each of size

In the first simulation study, we consider the weighed copula model from Equation 3.4. In particular, we take to be the Gaussian copula density with correlation parameter for which we take and to be the copula density of Model E1 with . This is a configuration for which likelihood-based inference is simply infeasible, owing to the nesting of two complex models requiring numerical integrals and inversion of functions. Figure 1 shows the results; the true values for each parameter are compared with their estimated values given by the trained NBE. It can be seen that there is a bit of variability, in particular for the parameters and we note that these are the parameters with the largest prior range. We can see that the NBE generally estimates the weighting parameter quite well, indicating that it is able to distinguish between the body and tail components of the copula. In addition, estimates of are quite accurate; this means that the two regimes of extremal dependence are reasonably well captured. In particular, of the data sets showing asymptotic dependence (i.e., ) were estimated to be AD, whereas of the data sets showing asymptotic independence (i.e., ) were estimated to be AI. Finally, estimates of exhibit the best results. Among negatively correlated data sets (i.e., ), were estimated to be negatively correlated, whereas of the positively correlated data sets were correctly identified to be positively correlated.

Fig. 1.

Assessment of the NBE when is the Gaussian copula with parameter is Model E1 with parameters and with weighting function for a sample size of The points highlighted in different shapes and colours refer to parameter configurations used for further diagnostics (see Figure 2)

It is important to assess the uncertainty of the NBE; we consider two different approaches for this task. For the first, a non-parametric bootstrap procedure is adopted. Specifically, we generate bootstrap samples by resampling with replacement. Then, for each bootstrap sample, the model parameters are estimated using the trained NBE. From these estimates, confidence intervals are constructed by taking the and quantiles of the samples. This is done for each parameter configuration and data from the test set. While there are no guarantees on the quality of the confidence intervals obtained through this procedure, it is computationally inexpensive to implement owing to the amortised nature of the NBE. This makes the bootstrap procedure a practical choice for exploring the uncertainty of these estimators. For the second approach, we train an additional estimator designed to target credible intervals. To do so, a composite quantile loss function targeting jointly low and high posterior quantiles is used instead. Precisely, this loss function is defined as for probability levels , and parameter estimators We train an NBE that outputs posterior quantiles for all parameters and probability levels, and note that the neural network is constrained in such a way that quantiles are properly ordered; see more details in Sainsbury-Dale et al. (2025). With this estimator, herein referred to as the neural interval estimator, we are able to approximate marginal central credible intervals by taking and setting We evaluate the performance of each approach by computing coverage probabilities, with the results presented in Table 1. The bootstrap procedure leads to lower than nominal coverage rates, with the extent of the miscalibration linked to the quality of estimates shown in Figure 1: the best coverage is obtained for the parameters showing a better correspondence between true and estimated values, such as or The parameters and exhibit more bias in their estimation, which is reflected in worse coverage rates. On the other hand, the neural interval estimator is much better calibrated. In the context of NBEs, bootstrap-based confidence intervals have been used extensively to account for the estimation uncertainty; however, to the best of our knowledge, their coverage rates have not been systematically explored.

Table 1.

Coverage probability and average length of the uncertainty intervals obtained via a non-parametric bootstrap procedure and via the neural interval estimator averaged over 1000 models fitted using a NBE (rounded to 2 decimal places)

Parameter	Bootstrap procedure		Interval estimator
	Coverage	Length	Coverage	Length
	0.80	0.27	0.97	0.57
	0.36	3.12	0.97	11.84
	0.53	3.42	0.96	10.32
	0.67	0.63	0.97	1.67
	0.70	1.12	0.97	2.54

We also explore whether the apparent bias in parameter estimates leads to bias in dependence quantities of interest. To do so, we consider parameter and data sets for which the NBE severely under- or over-estimates at least one parameter and compare the empirical and model-based dependence measure from Equation 3.1 at several thresholds For this model configuration, model-based values are estimated using a Monte Carlo approximation with samples. The parameter configurations considered are highlighted with different shapes and colours in Figure 1, and the results for are shown in Figure 2. Despite some parameters being greatly under/over-estimated, the dependence structure is still well captured overall, apart from the configuration shown in blue for which is massively over-estimated. Given that this is the parameter that controls the limiting extremal dependence structure of the data, directly determining the value of this result is not very surprising. Finally, from the bootstrap samples obtained previously, we compute coverage probabilities of confidence intervals for at levels these are shown in Table 2. As can be seen, the true value for is within the confidence intervals in at least of the time, suggesting that this derived feature of the models is relatively well estimated and better calibrated than the individual parameters when their estimates obtained by the NBE exhibit bias and display poor coverage properties.

Fig. 2.

Empirical (in grey), true (in pink) and model-based for the fitted weighted copula model with six different parameter configurations (in colour) for The 95 confidence bands, representing uncertainty in the empirical estimates, were obtained by boostrapping

Table 2.

Coverage probability and average length of the confidence intervals for at levels obtained via a non-parametric bootstrap procedure averaged over 1000 models fitted using a NBE (rounded to 2 decimal places)

	Coverage	Length
	0.86	0.09
	0.82	0.11
	0.79	0.12

In the second simulation study, we consider Model W with the priors mentioned in Section 4.1. Since this model is suitable for censored data, we consider a variable censoring level drawn from the prior given in Section 4.1. Figure 3 shows the performance of the trained NBE; estimates are quite accurate overall, but with some bias for large close to the prior support upper bound. We observe from the right panel of Figure 3 that the two regimes of extremal dependence are well captured; in particular, of the samples exhibiting asymptotic dependence (i.e., ) were estimated to be AD, while of samples exhibiting asymptotic independence (i.e., ) were correctly estimated to be AI.

Fig. 3.

Assessment of the NBE for Model W with parameters for a sample size of The points highlighted in different shapes and colours refer to parameter configurations used for further diagnostics (see Figure 4)

As with the first study, we assess the uncertainty of the NBE through non-parametric bootstrap and by training a neural interval estimator. Coverage probabilities of the uncertainty intervals and their average length are shown in Table 3. The coverage probabilities are once again better using the trained neural interval estimator, with wider intervals on average. Figure 4 shows the extremal dependence measure from Equation 3.1 computed as a further diagnostic for the four parameter configurations highlighted in Figure 3 at several thresholds where is the censoring level. Despite the under/over-estimation by the NBE, e.g., shown by the vector of parameters in red, the extremal dependence behaviour is well captured. This is further supported by the fairly good coverage probabilities of confidence intervals for at levels Coverage probabilities, reported in Table 4, are computed from estimates obtained from new data sets for 1000 parameter configurations, each computed with a fixed censoring level As a final diagnostic, we compare the joint tail behaviour along different rays. Transforming the model variables, and into standard exponentially distributed variables, and respectively, we compute the joint probability and its empirical counterpart for two different rays and The results for three parameter configurations exhibiting asymptotic independence are given in Figure 5. There is a very good agreement between the estimated and empirical joint probabilities for both rays considered, supporting the efficacy of the trained NBE.

Table 3.

Coverage probability and average length of the uncertainty intervals obtained via a non-parametric bootstrap procedure and via the neural interval estimator averaged over 1000 models fitted using a NBE (rounded to 2 decimal places)

Parameter	Bootstrap procedure		Interval estimator
	Coverage	Length	Coverage	Length
	0.76	3.59	0.96	6.68
	0.84	0.49	0.98	0.81

Fig. 4.

Empirical (in grey), true (in cyan) and model-based estimated via the NBE with four different parameter configurations (in colour) for where is the corresponding censoring level. A comparison with model estimated via censored maximum likelihood inference is given in pink. The 95 confidence bands, representing uncertainty in empirical estimates, were obtained by boostrapping

Table 4.

Coverage probability and average length of the confidence intervals for at levels obtained via a non-parametric bootstrap procedure averaged over 1000 models fitted using a NBE (rounded to 2 decimal places)

	Coverage	Length
	0.91	0.06
	0.89	0.09
	0.88	0.09

Fig. 5.

Empirical (in grey), true (in pink) and model-based estimated via the NBE with three different parameter configurations (in colour) for where is the corresponding censoring level. The 95 confidence bands, representing uncertainty in empirical estimates, were obtained by boostrapping

Comparison with censored maximum likelihood estimation

Finally, we compare estimates obtained by the NBE and those obtained by the censored maximum likelihood estimation (CMLE) procedure, using the same censoring scheme as outlined in Wadsworth et al. (2017). We first compute the CMLE for the four parameter configurations highlighted in Figure 3, and the corresponding estimates for for Results based on the CMLE, shown by the pink lines in Figure 4, are almost identical to those obtained using the NBE. Then, from the assigned prior distributions, we generate 5 different parameter vectors , censoring levels with corresponding data sets from Model W, each with a sample size of 1000. We then simulate data sets from each configuration 100 times. Figure 6 shows boxplots comparing the sampling distribution of the estimators and the corresponding for two parameter vectors: with censoring level and with (rounded to 2 decimal places). The remaining three cases are given in Section S2.2.1 of the Supplementary Material. Estimates given by the NBE are slightly more biased, but often less variable than the CMLE. Despite this small bias, estimates provided by the NBE are still relatively accurate whilst being much faster to obtain. In particular, the NBE took on average 0.676 seconds to evaluate, whereas the censored MLE took 183 seconds on average—a 225-fold speed-up.

Fig. 6.

Comparison between parameter estimates given by CMLE (orange) and by NBE (red), and corresponding for 100 samples with The true parameters are given by the red line. (a) with censoring level and (b) with censoring level

To assess the effect of treating the sample size n and/or censoring level as a random variable, we perform a similar study considering fixed censoring level with either fixed or variable sample size. The results are presented in Sections S2.2.2 and S2.2.3 of the Supplementary Material, respectively. Although the overall findings are similar, slightly higher coverage probabilities and shorter intervals, are obtained when n and are fixed.

Model selection assessment

Here we investigate the performance of the NBC for model selection, using the procedure outlined in Section 2.2. For each sampled model m during the training step, a data set is generated with a random sample size n, a random censoring level and a random vector of model parameters using the priors defined in Section 4.1. We demonstrate the performance of the trained NBC for the four models mentioned in Section 3.3, with censoring at level . The neural network architecture used for the model selection procedure is given in Table S12 of the Supplementary Material. We have compared all possible pairs among the four models, resulting in , and all models simultaneously, resulting in . Each model is selected with probability 1/K, and the sample simulated from that model is of size The class which has the highest output probability is the assigned model for the data set. We also compare the NBC with model selection via the Bayesian information criterion (BIC). This requires optimisation of the CMLE, which is time-consuming. However, we wish to explore how well the NBCs perform in comparison to established likelihood-based methods.

Figure 7 shows the assessment of the NBC when For this study, the proportions of correctly identified models (i.e., the models with the highest posterior probability matching the one from which data were simulated) with the NBC are always above with an average of whilst through BIC these are above with an average of Aside from the cases where the choice is between Model W and Model E2 or between Model HW and Model E2, the NBC and the likelihood-based BIC perform comparably, with the selection procedure through the NBC being quicker than using the BIC as no likelihood evaluation is needed. Finally, the uncertainty of the NBCs is assessed through a bootstrap procedure. Given samples, the confidence intervals for the proportions of correctly identified data sets are computed for each model and NBC; these results are reported in the middle bar plots of Figure 7.

Fig. 7.

Proportion (in ) of correctly identified data sets when through the NBC (middle) and through BIC (right) for the six pairs of models considered. The true counts of data sets generated from model (red) and from model (orange) are given in the left bar plot. The confidence intervals for the proportions of correctly identified data sets by the NBC are given in brackets

Results for the case when are displayed in Figure 8. Similarly to the previous case, the uncertainty of the NBC is assessed via a bootstrap procedure with bootstrap samples; the confidence intervals for the proportions obtained by the NBC are shown in the middle bar plot. The resulting proportions achieved through the NBC are all above with an average of whereas via the BIC they are above with and average of The proportions obtained via the BIC are generally lower than the ones obtained with the NBC apart from Model W, which indicates that the NBC correctly identifies the data sets more frequently than the BIC, and in a quicker way.

Fig. 8.

Proportion (in ) of correctly identified data sets when through the NBCs (middle) and through BIC (right). The true counts of data sets generated from models (red), (light red), (orange) and (light orange) are given in the left bar plot. The confidence intervals for the proportions of correctly identified data sets by the NBC are given in brackets

To assess the impact of the sample size on the performance of the NBC, and how this compares with the BIC, we have conducted a similar study, drawing 500 model indices, and simulating with a range of sample sizes The results are presented in Section S3.1 of the Supplementary Material. This shows that, across the training range, , the NBC outperforms BIC for small n, while the BIC is slightly better for larger n. Outside of the training range, at , the NBC still exhibits good performance. Interestingly the BIC is worse for than for . This non-intuitive result may reflect sampling variability, numerical problems, or convergence to local rather than global minima; however, further investigations did not reveal any convincing explanation for the reduced performance of the BIC for larger n, and the computationally-intensive nature of the likelihood-based methods means that it is hard to conduct more repetitions to confirm such conjectures.

Misspecified scenarios

We now examine the performance of the model selection and subsequent parameter estimation procedure in a misspecified scenario, where the underlying data do not come from one of the models considered. We consider two different situations and, in both cases, model selection is first done using the NBC, followed by estimation of the model parameters using the NBE trained for the selected models. A comparison with classical model selection and inference tools is also provided.

For the first study, we generate 100 samples, each with from a Gaussian copula with correlation parameter and consider a censoring level of The proportion of times each model was selected through the NBC and BIC is given on the top of Table 5, and the proportion of AD and AI samples identified by the NBE and CMLE is on the bottom. Model HW is the most suitable one according to either selection procedure, with the NBC selecting it of the time and the BIC of the time. This is to be expected given the nature of the underlying data and the model assumptions, i.e., we are assuming that follows a Gaussian copula in Model HW, such that the Gaussian copula arises as a limiting special case of the model as ; this implies that Model HW is the least misspecified model in this case. Both inference procedures are able to capture the correct extremal dependence structure. In particular, according to the NBE and CMLE, and of the samples, respectively, exhibit AI, which is in agreement with the underlying data since Gaussian data are known to be AI. According to the NBE, the two samples identified as coming from Model E1 exhibit AD, with estimated values (and corresponding confidence intervals) and For the CMLE, three of the samples identified as coming from Model W are AD; for these, the estimates for are close to 0, specifically and As a further diagnostic, we compute estimates at three levels for the selected models, and compare with the true at each level. The results are given in Figure 9. The estimates obtained with either inference method are concentrated around the true value, indicating that both the NBE and CMLE are able to capture the true extremal dependence structure quite well. In Section S4 of the Supplementary Material we describe an individual example, where the proposed toolbox for model selection and inference is presented in detail. The results obtained through this individual example agree with the findings of this study.

Table 5.

Proportion of times each model was selected through the NBC and through BIC (top), and proportion of AD and AI samples identified by the NBE and CMLE (bottom). All the values are rounded up to 2 decimal places

Model	NBC	BIC
Model W	0.02	0.30
Model HW	0.88	0.69
Model E1	0.02	0.00
Model E2	0.08	0.01

Method	AD	AI
NBE	0.02	0.98
CMLE	0.03	0.97

Fig. 9.

Model-based estimates of given by the NBE (red) and by the CMLE (orange) for levels and for 100 samples of (a) a Gaussian copula with correlation parameter and and of (b) a logistic distribution with dependence parameter and For both cases, the true value is given by the dashed red lines

For the second case, we generate 100 samples, each with from a logistic distribution (Gumbel 1960) with dependence parameter (the smaller the stronger the extremal dependence) and consider a censoring level of The proportion of times each model was selected and the proportion of AD and AI samples identified by the NBE and CMLE are given in Table 6. For this case, the model selection procedures differ, with the NBC selecting Model HW of the time, and the BIC selecting Model W as the most suitable one in of cases. Additionally, the CMLE correctly identifies all of the samples to be AD, whereas the NBE misclassifies 16 of the samples fitted with Model HW as being AI. For these 16 samples, the estimated values for are near 0.5 (precisely ), the boundary point for AI, in 12 samples. Moreover, in all 16 samples, the boundary point is within the uncertainty intervals. The estimates for at three levels are shown in Figure 9. Contrarily to the previous case, the NBE somewhat under-estimates the true of the logistic distribution, while the CMLE under-estimates it slightly only at higher levels. Moreover, the estimates provided by the NBE exhibit higher variability than the CMLE. As with the first specification in the section, an individual example is presented in Section S4 of the Supplementary Material. Similarly to this study, Model HW is selected as the most suitable one to fit the data and seems to under-estimate the true for

Table 6.

Proportion of times each model was selected through the NBC and through BIC (top), and proportion of AD and AI samples identified by the NBE and CMLE (bottom). All the values are rounded up to 2 decimal places

Model	NBC	BIC
Model W	0.18	0.42
Model HW	0.79	0.21
Model E1	0.02	0.08
Model E2	0.01	0.29

Method	AD	AI
NBE	0.84	0.16
CMLE	1.00	0.00

Case study: changes in geomagnetic field fluctuations

Data and background

The behaviour of the sun and the consequences of its interaction with the Earth’s magnetic field and atmosphere are known as space weather events. Examples include phenomena such as the auroras, often known as Southern and Northern lights, or solar storms. These events can cause large fluctuations in the geomagnetic field, leading to geomagnetically induced currents (GICs), which are electrical currents generated at the surface of the planet by rapid changes in the magnetic field. Furthermore, GICs can cause disruptions on power grids, communication systems, railway systems, and other critical infrastructures. Thus, the modelling of extreme solar activity can help prevent the impacts of GICs.

Following Rogers et al. (2020), we use the rate of change of the horizontal component of geomagnetic field, which is available through the SuperMAG interface (Gjerloev 2009), as a measure of the magnitude of GICs. In particular, we apply the proposed toolbox to select and make inference on the pairwise extremal dependence structure between measurements at three pairs of locations in the northern hemisphere: two in Greenland and one in the east coast of Canada (see Table 7)—to assess whether a large magnitude of GICs occurring in one location has an effect on another location. We take daily maximum absolute one-minute changes in which results in 7572 complete observations. However, since the estimators are trained for sample sizes between 100 and 1500, we take a subset of the data set with retaining every 5^th observation in order to reduce the temporal dependence present in the data, and truncate the resulting data set to 1500 observations by removing the last few. Figure 10 shows the scatterplots of pairwise daily maxima absolute one-minute changes in between the pairs of locations considered. We first transform the data to uniform margins using the semi-parametric approach of Coles and Tawn (1991) with a generalised Pareto distribution (GPD) fit to the tail of the data. Thus, the cdf of each margin is estimated via

where is the empirical distribution function, is the estimated probability of exceeding a selecting high threshold r, and and are the estimated scale and shape parameters of the GPD, respectively. In particular, with r as the marginal quantile of the data, we have for SCO, for STF, and for STJ.

Table 7.

International Association of Geomagnetism and Aeronomy (IAGA) code, and location of the observatory for the three locations considered

IAGA code	Observatory (Country)	Latitude	Longitude
SCO	Scoresby Sund 2 (Greenland)	70.48
STF	Sdr Stromfjord (Greenland)	67.02
STJ	St. John’s (Canada)	47.60

Fig. 10.

Daily maxima absolute one-minute changes in measurements between three pairs of locations: (SCO, STF) on the left, (SCO, STJ) in the middle, and (STF, STJ) on the right

Statistical inference

We are interested in modelling the joint extremal behaviour of between each pair of locations, so we focus on the four models mentioned in Section 3.3, and censor the non-extreme observations; we do so by taking as the censoring threshold. We note, however, that we have considered a range of censoring levels and summarise these results in Section S5 of the Supplementary Material. We start by applying our NBC to select the best model and estimate its parameters through the corresponding trained NBE; the results are shown in Tables 8, 9 and 10 for pairs (SCO, STF), (SCO, STJ) and (STF, STJ), respectively. The fit of the preferred model is then assessed by comparing the estimated dependence measure with its empirical counterparts for levels As extra comparisons, the estimated measures for the model with the second highest posterior probability are also obtained, and a Bayesian model averaged approximation of is computed, by setting where and are the posterior probabilities and estimated for Finally, a comparison with the results obtained through censored MLE is also provided; the results for are shown in Figure 11, where the confidence bands for empirical estimates are obtained via block bootstrap with a block length of 10 to reflect the remaining temporal dependence present in the data.

Table 8.

Model selection procedure obtained through the NBC for censoring level and parameter estimates given by the trained NBE for pair (SCO, STF). The results through censoring MLE and BIC are given in the bottom table. All the values are rounded to 3 decimal places

Model			Extremal dependence
W			AD
HW			AI
E1			AI
E2	0.078		AI

Model	BIC		Extremal dependence
W	226.785		AD
HW	215.897		AI
E1	214.556		AI
E2			AI

Table 9.

Model selection procedure obtained through the NBC for censoring level and parameter estimates given by the trained NBE for pair (SCO, STJ). The results through censoring MLE and BIC are given in the bottom table. All the values are rounded to 3 decimal places

Model			Extremal dependence
W	0.029		AI
HW	0.072		AI
E1			AI
E2			AI

Model	BIC		Extremal dependence
W	390.990		AI
HW			AI
E1	396.119		AI
E2	393.674		AI

Table 10.

Model selection procedure obtained through the NBC for censoring level and parameter estimates given by the trained NBE for pair (STF, STJ). The results through censoring MLE and BIC are given in the bottom table. All the values are rounded to 3 decimal places

Model			Extremal dependence
W	0.308		AI
HW	0.019		AI
E1			AI
E2			AI

Model	BIC		Extremal dependence
W	398.908		AI
HW			AI
E1	401.390		AI
E2	409.988		AI

Fig. 11.

Empirical (in grey) and model estimated via the NBE for for the models with the two highest posterior probabilities. For each pair, the estimated for the selected model is given by the thick line, followed by its estimate for the model with the second highest probability given by the dashed line. The BMA approximation of is given by the pink line. Note that this may overlap with the estimated for the model with the highest posterior probability. Estimated model for the selected model through BIC is given by the purple line. The 95 confidence bands were obtained by block boostrapping

For pair (SCO, STF), Model HW is clearly the selected model with a probability of 0.922 followed by Model E2 with probability 0.078. As can be seen by the estimates given by the parameters of interest, and respectively, both models indicate the presence of asymptotic independence in the data. The same is not true for Model W. It can be seen on the left panel of Figure 11 that models HW and E2 (in blue and orange, respectively) seem to capture the extremal dependence of data well. According to the BIC, however, Model E2 is the most appropriate to fit this pair. Like the NBE estimates, the CMLE estimates for this model indicate the presence of asymptotic independence, which is in agreement with the estimates obtained by the NBE and CMLE for models HW and E1. Furthermore, the estimated given by the CMLE (in purple) almost overlaps with the obtained with Model E2. When censoring below a range of thresholds, Model E2 or HW were consistently the preferred choices, both indicating asymptotic independence.

Model E2 is clearly selected as the best model to fit pair (SCO, STJ) with a posterior probability of 0.9. However, as shown by the middle panel of Figure 11, the estimated measure (in orange) is somewhat below the empirical estimate for higher levels u. In turn, the estimated measure given by Model HW (in blue) is closer to its empirical counterpart further in the tail, suggesting a better fit of the dependence structure in the limit. Although small, the probability for Model HW given by the NBC is the second highest, followed closely by Model W. For this pair, all four models indicate the presence of asymptotic independence through the estimates given by the NBE and CMLE. Contrarily to the NBC, BIC selects Model HW as the best model to fit the data, with a very small difference compared to model W. The estimated highlighted in purple, is also able to capture the extremal dependence well, particularly further in the tail. Similarly to the first pair, Model HW or E2 were selected as the preferred fit for pair (SCO, STJ) for a range of censoring levels, both models indicating asymptotic independence.

For the final pair (STF, STJ), Model E2 is again selected as the preferred model with a probability of 0.672. Model W closely follows with a posterior probability of 0.308. However, Model E2 has the highest BIC value, indicating that, in the likelihood framework, any of the other models would provide a better fit for this pair. The right panel of Figure 11 shows that Model E2 (in orange) leads to an estimate below the empirical for all levels u considered, whilst Model HW (obtained by CMLE and shown in purple) gives an estimate slightly above the empirical value of On the other hand, Model W (in green) best captures the extremal dependence structure over most of the range. Despite this, all models indicate asymptotically independent variables; this is also in agreement with the estimates obtained by censored maximum likelihood inference. Interestingly enough, the estimated value of parameter of Model E1 given by either approach is very close to 1. When considering different censoring thresholds, the NBC consistently chose either Model HW or E2 as the best fit for the pair (STF, STJ), both models indicating asymptotic independence across all censoring levels.

A few conclusions can be drawn from these results. For instance, although the selected model may not always align most closely with the empirical , a good representation can still be achieved if the model with the second highest probability is considered instead; furthermore the majority of estimates remained in the pointwise 95% confidence intervals for the empirical estimates. When considering a BMA approximation of the influence of the model selected through the NBC is clear. Additionally, with the proposed toolbox, we are not only able to infer about some model characteristics, but also able to assess the sensitivity to the censoring threshold by considering a range of different levels In particular, from the results obtained for the range of considered (see Section S5 of the Supplementary Material), we can see that, for lower censoring levels, Model HW is clearly the preferred one to fit the three pairs, with posterior probabilities above 0.9 in all cases. This changes for censoring levels or for pair (SCO, STF), for which Model E2 is the most suitable one to fit the data according to the NBC. In all cases, the selected models consistently indicate the presence of asymptotic independence in each pair. This sensitivity analysis to the censoring level is computationally expensive in a likelihood-based inference method, where the likelihood needs to be evaluated for all considered models across various censoring levels. Likelihood estimation can indeed be about 7300 times slower than using the NBE, especially for Models E1 or E2, where the code is not as efficient as for Models W or HW. This leads to the need for choosing a censoring level before the analysis, which might end up not being the most suitable one for the underlying application.

Conclusion and discussion

In some situations, the likelihood function of a model may be available but its evaluation may be computationally costly. In multivariate extremes, this is often due to the need for numerical inversion of functions, and numerical integration. This computational burden in the inference process poses limitations on the use of certain models in practice since modelling usually entails consideration of different candidate models and threshold levels. In this paper, we exploited the use of neural Bayes estimators, a modern likelihood-free approach which uses neural networks, to perform inference on the vector of model parameters. In particular, we focused on two types of models which are available in the multivariate extremes literature: a weighted copula model, which is able to represent both the body and tail regions of a data set, and models based on random scale constructions, which are flexible enough to capture the two regimes of extremal dependence, interpolating between the two in the interior of the parameter space. When likelihood evaluation is no longer required, model selection criteria such as the AIC or BIC are now unavailable for determining the best-fitting model. We overcame this by proposing a neural Bayes classifier which allows us to choose the most suitable model from candidates models in a fast and effective manner.

For the models where it is feasible—albeit relatively slow—to evaluate the likelihood function, we compared the performance of the neural Bayes estimators and NBC with (censored) maximum likelihood inference and BIC, respectively. Through simulation studies, we have shown that NBEs perform quite well overall, though they tend to be slightly more biased than the (censored) MLE in some cases; however, they also sometimes exhibit lower variability. Moreover, when estimating tail quantities of interest, such as the extremal measures from Equation 3.1, we did not find the bias to be problematic, as the overall dependence structure appeared to be well captured with the trained NBEs anyway. Nevertheless, the bias of the estimates is reflected by lower coverage probabilities when using bootstrap-based credible intervals. Whilst coverage probabilities significantly improved when we trained a neural interval estimator, obtaining a value close to nominal, it would be ideal to also obtain such values with bootstrap-based intervals as we would only need to train the estimator once using a single loss function. Despite this, since the neural Bayes estimates are considerably faster to obtain than the (censored) MLE, which may also sometimes converge to local rather than global minima, and the extremal dependence structure is generally well captured, the methodology may be considered preferable overall.

Given the symmetry of all models considered, an estimator that is invariant to permutations of the vector components would be desirable. However, this is not achieved with the DeepSets architectures mentioned in Section 2.1, which results in different estimates when considering and . We investigated this issue by creating an alternative estimator that is the average of the NBE obtained with inputs and , but the results were almost indistinguishable from the basic approach presented.

When comparing the NBC with the BIC for model selection, both procedures performed similarly with candidate models. However, when and , the NBC was able to correctly categorise more data sets than the BIC for all considered models, except Model W. As shown in Section S3.1 of the Supplementary Material, this conclusion also holds for smaller sample sizes. Interestingly in Section 5, the BIC and NBC often displayed large disagreements in the selected model. While all selected models generally performed adequately, it could be of interest in further work to explore the effect of being on the boundary of our set of simulated sample sizes. We also demonstrated the possibility of using the NBC as part of Bayesian model averaging-like approach to estimate measures of interest such as . However, full Bayesian model averaging requires complete posterior distributions rather than individual estimates. Combining the NBC with neural posterior approximations such as normalising flows (Radev et al. 2023) would be beneficial for amortised BMA, and presents an interesting area for future research.

We have shown that the proposed toolbox for model selection and inference performs well—though not always as accurately as classical likelihood inference—for misspecified scenarios where the data set does not originate from one of the models considered in this paper, with the extremal dependence structure being generally well captured. In particular, through repeated simulation, we showed that the NBC and NBE captured the true extremal dependence structure the majority of the times, especially for asymptotically independent data. Further investigation on the performance of NBEs in misspecified scenarios is an important future line of research. An alternative and interesting line for further research is to adapt the current framework to incorporate an unobserved model class into the model selection framework when there is sufficient evidence that the data is not generated by any of the K candidate parametric models.

In all our examples, we assumed the parameters to be uniformly distributed a priori. However, we note this implies non-uniform priors in alternative parameterisations and we did not assess the effect of this choice on the performance of the trained NBE. Despite this, for studies involving the weighted copula model from Section 3.4, we found that reparameterising some of the model parameters (see Section S2.1 of the Supplementary Material) helped the neural network learn about them in the training step of the NBE. More specifically, the assessment of the NBE showed less variability and higher coverage probabilities (with narrower range) of the uncertainty intervals for each model parameters obtained with bootstrap-based intervals. Additionally, the (censored) MLE only depends on the likelihood function as opposed to the NBE, which relies on the choice of a prior. While the (C)MLE is the most used likelihood-based inference method in the extremes literature, it would be of interest to incorporate the same prior belief and obtain a (censored) maximum a posteriori (MAP) estimate instead. Investigating and comparing the performance of the (C)MAP with the NBE in our bivariate copula context presents an interesting avenue for further research.

We restricted our analysis to the bivariate setting. Each of the random scale models described in Section 3.3 could be expanded to higher dimensions; however, this may not be as useful since the models considered are only suitable when all the variables are asymptotically dependent or asymptotically independent. On the other hand, the dynamic mixture model could usefully be extended to higher dimensions, and the benefits of NBEs may be even clearer in this setting as the complexity of the likelihood increases with dimension for this model.

The proposed toolbox has potential to be extended to other settings, such as online learning, change-point detection, and trend estimation problems. In particular, recent work has explored deep learning techniques for change-point detection (see, e.g., Li et al. 2024). Within our framework, a prior on the number and position of change-points could be used and a neural estimator could then be trained under a suitable loss function. This would allow estimating change-points simultaneously with the other model parameters. For capturing trends, we could incorporate covariates into the analysis. However, to amortise over covariate values, a suitable prior model for the covariates would be required. Depending on the application, this might not be straightforward to specify.

Supplementary information

Supplementary Material for ‘Neural Bayes estimation and selection of complex bivariate extremal dependence models’: File containing additional results and figures. (supplementary.pdf)

Supplementary Information

Below is the link to the electronic supplementary material.

Author Contributions

All authors contributed to the development of the core ideas of the paper. Raphaël Huser contributed primarily to the inference methodology, while Jennifer Wadsworth contributed substantially in discussions on the simulation and case study analysis, and provided code to implement the models of Engelke et al. (2019). Lídia André developed all other code and performed all the simulation and case studies. The first draft of the manuscript was written by Lídia André, and all authors commented on previous versions. All authors read and approved the final version of the manuscript.

Funding

This work was fully supported by EPSRC grant number EP/S022252/1 and partially by the Fonds de la Recherche Scientifique–FNRS, Belgium (grant number T.0203.21).

Data Availability

The ground magnetometer data were provided by SuperMAG (available from https://supermag.jhuapl.edu/).

Code Availability

Code supporting the findings of this study are available on https://github.com/lidiamandre/NBE_classifier_depmodels.

Declarations

Competing interests

The authors declare no competing interests.

Ethical Approval and Consent to Participate

Not Applicable

Consent for Publication

Not Applicable