Abstract
In this paper, we start by reviewing exchangeability and its relevance to the Bayesian approach. We highlight the predictive nature of Bayesian models and the symmetry assumptions implied by beliefs of an underlying exchangeable sequence of observations. By taking a closer look at the Bayesian bootstrap, the parametric bootstrap of Efron and a version of Bayesian thinking about inference uncovered by Doob based on martingales, we introduce a parametric Bayesian bootstrap. Martingales play a fundamental role. Illustrations are presented as is the relevant theory.
This article is part of the theme issue ‘Bayesian inference: challenges, perspectives, and prospects’.
Keywords: bootstrap, parametric bootstrap, predictive inference, score function
1. Introduction
Exchangeability forms one of the key starting points for the Bayesian approach, with many articles commencing along the lines ‘Assume the observations to beexchangeable’. The existence of a prior distribution, following such an assumption, has provided a foundation for the Bayesian paradigm (see ch. 4, Bernardo & Smith [1]). The existence of the prior from the representation theorem was established originally by de Finetti [2], for binary outcomes, and later for more general outcomes by Hewitt & Savage [3].
The exchangeable assumption on variables has the implication that the order of the labels becomes irrelevant. Consequently, forms an infinite exchangeable sequence if, and only if the joint distribution of is invariant under permutations, i.e.
| 1.1 |
for all permutations on and for all . Hewitt & Savage [3] referred to this as the joint density being symmetric, in contrast to de Finetti who talked about the exchangeability of the variables. Under the exchangeable assumption, de Finetti [2] showed, for variables, that there is a representation of the joint density having the form
for some (prior) distribution function . For Bayesians, the prior forms the model, and so the sequence is set by the corresponding density function , assumed to exist, and defined on . Then ; i.e. the given are independent Bernoulli random variables. Hewitt & Savage [3] extended this result to more general types of symmetric density.
The existence of the prior from the representation theorem was of great importance to the Bayesian argument. See, for example, Cifarelli & Regazzini [4] and Bernardo & Smith [1]. It motivated Bayesian analysis through the simple assumption of exchangeable observables. The breakthrough paper of Lindley & Smith [5] was based on the idea of exchangeability, as noted in the opening line from the abstract, 'The usual linear statistical model is reanalyzed using Bayesian methods and the concept of exchangeability'.
Exchangeability is seen to be equivalent to saying that the order in which the data arrives makes no difference to inference; indeed, the probability model for the observations is symmetric. The predictive density follows under the usual conditioning process, i.e.
Clearly, from the definition of exchangeability, the arrangement/ordering of the conditioned becomes irrelevant.
It is also possible to view the predictive using an update of the prior , that is
| 1.2 |
where
| 1.3 |
is the posterior distribution, with the normalized likelihood function for data point .
Exchangeability has been viewed as a property on the data; creating a dependence structure where in reality there could be none. To see the argument, suppose the experimenter is to predict the first outcome ; in order to make a decision, for example. Write this predictive as . Once has been viewed, the experimenter might then wish to predict . It would be an obstinate experimenter who persisted to predict with , since it would normally be assumed that witnessing would somehow revise the prediction for . Indeed, the rationale for a statistical analysis and learning from data is that the observation provides information about the underlying process or population leading to improved prediction.
So write the revised predictive as . It is easy to see how such a procedure progresses; one eventually predicts using the revised density . This results in a joint predictive density for the forthcoming observations to be
If indeed the order is assumed not to matter; reflecting the notion that the observations could be independent and identically distributed (i.i.d.), then it would be required that be symmetric in the arguments and hence the existence of the prior-likelihood representation and the Bayesian approach. We write ‘could be’ here to reflect the notion that the data do not explicitly yield how they arose and the leap from the mathematics to the day to day practical data analysis relies on assumptions.
The above motivation de-mystifies Bayes. It is seen as a learning approach, updating beliefs as data arrive. It is no less and no more about the modelling a dependence between the observables arising as a consequence of revising predictives given observations. The crucial point is that it is the sequence of predictive distribution functions that depend on the observations rather than the next observation having some physical dependence on what has previously been seen. By physical dependence between variables and it would be assumed there is a form for some function . In short, the exchangeable model is an appropriate learning model for i.i.d. data structures.
An early application of the predictive viewpoint for finite populations was developed in Roberts [6], Ericson [7] and later by Geisser [8,9]. This was extended to non-parametric predictive models in Lo [10] and Ghosh & Meeden [11]. Fortini et al. [12] explored concepts of predictive sufficiency and Fortini & Petrone [13,14] discuss the construction of a range of popular exchangeable non-parametric priors through a sequence of predictive distributions, motivated through a predictive de Finetti’s representation theorem.
There are a number of articles that focus on the construction of predictive distributions derived from representation theorems and relying on notions of exchangeability and partial exchangeability. Here, we mention Lijoi et al. [15,16], Favaro et al. [17] and De Blasi [18]. In such cases, and when the posterior is more complicated to derive, then full Bayesian inference can be achieved through these predictive distributions.
(a) . Prediction and model evidence
The predictive viewpoint is especially useful for understanding the plausibility of two or more competing theories (models) for the data, and the updating of such beliefs. We have seen how a hypothesis, such as a posited model , for the data is characterized by the joint predictive , which provides an empirical model score when evaluated on the observed data, as seen through the 1-step predictions, and updates, on a sequence of held-out data
Comparing two hypotheses (models), versus , can be done by considering the Bayes factor (Kass & Raftery [19])
obtained by the relative predictive scores on the observed sequence . This highlights that Bayesian model choice is in essence a comparison of predictive performance on observables.
With exchangeability, the total score for any model is invariant to the possible ways of constructing the held-out predictive sequence
noting the th element in the sum on the right-hand side has the form . It is then straightforward to show [20,21] that the predictive score is equivalent to exhaustive cross-validation averaged over all -choose- possible held-out datasets of size , for all , when using the log predictive, , as the cross-validation scoring rule. Moreover, Fong & Holmes [20] show this to be the unique scoring rule that ensures order coherency in the model evaluation, and this is irrespective of whether the model is true. Taken together, this highlights the usefulness of the predictive viewpoint and its focus on observables.
(b) . Generalizations
A concern could well be to what extent does a representation theorem cover different and more complex data structures than i.i.d. sequences, such as stationary sequences. A sequence is stationary if
for any and any . The representation of a stationary sequence due to Mitra [22] establishes the existence of a prior for which it is possible to write
See also Aldous [23] and Lijoi et al. [15,16]. The connection between stationary sequences and exchangeable sequences has been considered by Berti et al. [24], where the link involves conditionally identically distributed (c.i.d.) sequences. A c.i.d. sequence is such that
for all . That is, marginally, has the same distribution as for any .
For more complex data structures, the typical case in the modern data science era, there will be no representation theorem. Hence, some guiding principles are needed. The usual approach is to assign prior distributions to parameters of a likelihood function; to see Bayes as nothing more than likelihood times prior; see Robert [25]. While this is a simple plan, it is not motivated in a de Finetti sense. It is unclear that statistical uncertainty is being reflected in any meaningful way if the majority of prior distributions are constructed for convenience and attempted to be set up for objective reasons.
One attempt to cover more types of data structures is to use the notion of partial exchangeability. See for example, Camerlenghi et al. [26] who look at the problem from a predictive perspective. An interesting question is to what extent partial exchangeability extends to real data, such as covariate-driven non-homogeneous Markov time series with unknown order.
2. A new look at the Bayesian approach
Fong et al. [27] describe how Bayesian uncertainty can be interpreted through missing data. To make this notion concrete, in the case of i.i.d. observations from an infinite population, the missing data when a sample of size has been observed; i.e. , is the . Our interpretation of the Bayesian approach is to deal with this uncertainty by providing a conditional distribution for . More generally, the assumption behind most, if not all, statistical problems is that there is an amount of data, let us label it as , which if observed, would yield the problem solved. That is, the parameter to be estimated is done so exactly, or the decision to be made can be done so correctly. For example, in a clinical trial if the recruited population was so large that any resulting uncertainty of the treatment effect estimate was considered negligible. So for an observed dataset, say , the Bayesian approach is to construct a predictive distribution such that .
The traditional Bayesian approach can be understood from this perspective. To illustrate this, let us return to the i.i.d. observations and for any sample of size , there is a predictive density as in (1.2) and derived from the posterior (1.3). Then
Now use this predictive to generate a large sample which we will regard as for some suitably large . Once this has been obtained, we define , which is some functional of the data of interest. For example, it could be the sample mean. Doob [28], in his paper on the applications of martingales, establishes under extremely mild conditions, that if represents the posterior mean, then a.s. and is a sample from the posterior derived from the sample of size .
A less rigorous but intuitive construction is by noting the conventional Bayesian posterior on parameters of interest can be written as
leading to
| 2.1 |
where can be taken so large so as to yield no relevant uncertainty in the parameter of interest, , meaning in the context of a particular analysis for all practical purposes the conditional posterior can be replaced with a point estimate . This highlights the source of the uncertainty in the Bayesian posterior, i.e. the left-hand side of (2.1), which arises from the missing data in the population. The conventional Bayesian posterior is recovered by taking as the posterior mean. A numerical solution to (2.1) follows a data-augmentation Monte Carlo approach by simulating using the sequence of 1-step predictives and then picking off the resulting estimate . As shown in Fong et al. [27], the computational solution is trivially parallel and often much faster to run than Markov chain Monte Carlo (MCMC). This missing data view also points to extensions of traditional Bayes involving more general predictive machines for use in .
The Bayes posterior (2.1) has two components,
The usual Bayesian approach is to construct the predictive, , using a likelihood-prior mixture (1.2), that is fixed a priori, with inference focused on parameters of the likelihood. This ensures exchangeability and provides for a simple update via Bayes rule. The approach in Fong et al. [27] supports the replacement of with an estimate, , using an appropriate functional targeting any statistic of interest, not necessarily indexing a likelihood function, and a predictive built using all available information at time of inference, not necessarily involving a prior. For example, in this regard, it is perfectly valid and reasonable to consider uncertainty in estimates arising from the Bayes linear model (e.g. Lindley & Smith [5]) obtained in the limit of large data, without assuming that the true data generating process is a linear model. The conventional Bayesian approach conflates these two issues, that the estimate refers to parameters in the predictive model assumed to be true (see §3).
(a) . Martingales
In fact, martingales are important for this view of the Bayesian methodology. Consider the posterior mean conditional on , i.e.
where has been taken from the predictive . Then, clearly, and so for , the sequence of posterior means forms a martingale. Indeed, any mean functional would form a martingale as would the sequence of posterior distributions itself. Martingales are the important feature to ensure convergence of the parameter of interest.
Our argument, and promoted in Fong et al. [27], is that it is the martingale property which should be the determining criterion for the choice of and, more generally, for . That is, if we define and has been taken from a ‘predictive’ , then we require that .
Fong et al. [27] focus on non-parametric models including a sequence of non-parametric predictives using copulas. Specifically, they take
| 2.2 |
where the weights are of the form and is a partial derivative of a Gaussian copula function with correlation parameter . That is , which yields
The sequence forms a martingale when the comes from . The sequence commences at the distribution estimator from the observed sample . Convergence of to a random distribution function follows from Berti et al. [29]. The limit is regarded as a random draw from the martingale posterior distribution.
The use of copulas here should not come as any great surprise. Consider the Bayesian predictive density given by
which can be written as
The fraction part on the right is a copula density function, which can be written as
for some copula density . In most cases, this copula itself depends on the ; though in some conjugate Bayesian models, it only depends on the sample size . Indeed, the complexity of prior to posterior procedures can be rendered difficult due to the nature of an intractable copula when it does depend on . This motivated Hahn et al. [30] to adopt copula densities which only depended on the sample size and took the non-parametric copula distribution function
i.e. a sample size weighted combination of the independence copula and the Gaussian copula with correlation parameter . The non-parametric credentials are valid here since the Dirichlet process sequence of predictives arise when, instead of , we use the maximum copula, , which arises as .
In order to better understand the martingale posterior and to investigate the parametric counterparts, we reconsider the bootstrap techniques, notably the bootstrap and Bayesian bootstrap. This will help us better understand the separate dealings with statistical uncertainty.
3. The bootstrap and Bayesian bootstrap
The bootstraps are very insightful tools with which to understand how the frequentist and Bayesian approaches to dealing with uncertainty differ. Both start with the empirical distribution function , which puts mass at each of the observed data points.
The frequentist regards the sample of size ; i.e. as being the source of uncertainty, in that it is but one of many possible observed samples of size . The uncertainty in this sample, and the consequences of this, could be ascertained if multiple independent samples of size could be gathered. This is not possible as the true distribution is unknown; but the empirical distribution function is available. Sampling alternative datasets of size is therefore taken from the empirical distribution function . Multiple bootstrap samples can be obtained, for . From each of these bootstrap samples, a statistic of interest can be computed, and consequently the sampling variance of can be estimated.
The Bayesian bootstrap was introduced in Rubin [31] and was seen as a Bayesian version of the bootstrap introduced in Efron [32]. The reason why this comparison was made is because both generate random distributions of the form
For the Efron bootstrap, the random weights are based on multi-nomial sampling; . On the other hand, the Bayesian credentials for the Bayesian bootstrap is due to its connection with the Dirichlet process [33]. The Bayesian bootstrap generates random probability distributions which attach random weights to the set of observed data ; i.e.
| 3.1 |
where the have a Dirichlet distribution with all the parameters set to 1. This version of the bootstrap is very clear and that it belongs to the Bayesian non-parametric set of tools is also very clear. However, based on work done by Blackwell & MacQueen [34], which can be seen as a follow-up to the Doob [28] result, the Bayesian bootstrap can be viewed in an alternative and illuminating way.
Take a sample of size and construct the empirical distribution function, . Then sample and update the empirical distribution with the new sample. That is, if is taken from then construct the new empirical predictive distribution given by
Continue in this fashion; to get and and so on. The sequence forms a martingale and some elementary theory for martingales on distribution functions [24] indicates that with probability one converges to a random distribution . Since can only be of the form (3.1) it is only a matter of finding the weights attached to the . It is not difficult to establish the correct Dirichlet weights for them; see Blackwell & MacQueen [34] and Sethuraman [35].
Thus we see the Bayesian bootstrap is constructing a predictive distribution and it is easy to see the sequence is given by
| 3.2 |
with coming from , and . The sequence forms a martingale and this is easy to check. The limit can be sampled according to (3.1). Then (3.1) can be viewed as a ‘short-cut’ to avoid the sampling of the . The non-trivial differences between the two types of bootstrap is quite clear.
Given , and the missing data view from §2, we can proceed to make inference by picking off an estimate of any particular statistic of interest, . One particularly interesting statistic is the estimate of the parameters in a statistical model, , minimizing the self-information loss (log-likelihood), i.e.
With as the Bayesian bootstrap leads to the weighted-likelihood bootstrap and extensions thereof [36,–41]. As noted in §2, we see it as perfectly valid to think about the Bayesian uncertainty in the value of arising from a statistical model, , fit in the limit of large data, without assuming that the data were generated by this statistical model for some unknown setting of .
It is to be noted that the sequence in (2.2) can be seen as a smoothed version of (3.2), where the indicator function has been smoothed to the Gaussian copula. Also note that as it is that .
The frequentist is replicating statistics of sample size . The Bayesian is obtaining ‘true’ statistics via and acknowledging the uncertainty in any single through the posterior it generates. It is this latter view of the Bayesian handling of uncertainty that is of interest to us. It is our intention to extend the family of Bayesian bootstraps and to use these to construct posterior distributions and adopting this as the fundamental Bayesian paradigm. That is, the construction of , namely , is the fundamental task while ensuring convergence of the statistics of interest, , and the sequence of predictive distribution. This can be guaranteed via the use of martingales.
Our understanding of the construction of is that it is prior free; it starts at the sample size . Instead of generating multiple replica datasets of size from this, along the lines of the frequentist bootstrap, we use it to generate the missing data along the line of the Bayesian bootstrap and the construction of a posterior distribution. To acknowledge the fusion here,we label the final output as a form of frequentist posterior; as we can take to be the product of for , where the would be the maximum-likelihood estimator based on .
4. A parametric Bayesian bootstrap
Consider the data model . The parametric bootstrap introduced in Efron [42] replaces the empirical distribution function with the plug-in density estimator, . The reasons for this become straightforward if the model is assumed to be correct, or is approximately so. Here, the is some functional of the data, such as the mle. In all other respects, the parametric bootstrap mirrors that of the non-parametric one. So a bootstrap sample is taken i.i.d. from . For each sample, a new mle estimator is obtained as and the collection of such can be used to estimate the sampling distribution and specifically the variance of .
Efron observed that the distribution of the resembles closely a possible posterior distribution and advocated it could be used in some importance sampling procedure to obtain samples from a posterior. It cannot be regarded as a posterior itself as it is not derivable from a prior and the more important point is that it is a distribution for the estimator, not for the true parameter value.
The connection and differences between the bootstrap and Bayesian bootstrap have shown us that a different sampling strategy from implements our Bayesian approach. The Bayesian version assumes further the is an unbiased estimator for the true value . We then sample from the data model using the plug-in which is then updated to using , and so on. In short, given with estimator , we perform
The form a martingale and provided the variances do not grow but converge then a random exists and can be seen as a sample from a ‘parametric martingale’ posterior distribution .
It is worth at this point considering a simple example. Suppose we have a normal location parameter model for which the data model is where is unknown and is known. Then from a sample of size we get the which is an unbiased estimator for . We then sample from and get
and so on. This is a martingale for and to investigate convergence, we consider the . Now writing , where is a standard normal random variable, we get
resulting in being a normal random variable with mean and variance where . This leads to the frequentist posterior as a probability measure directly on the unknown as approximately .
In general, we assume we can write
for some function , where comes from . Hence, for the martingale, we require
and so the resembles or acts similarly to a score function. Note that it is easy to compute the variance of since it would be equal to
This finding motivates a potentially simpler idea, which is to use the score function from the data model; i.e. define . Then we can use the sequence of estimators as
where comes from . Here, would act as a step size and the updating here resembles a gradient descent algorithm. The variance of would now be
which if is of order then will be of order , which is usual for parametric posterior distributions.
The idea here for updating the parameter is as follows. If there is a procedure for updating the parameter estimator as real data arrive; e.g. the mle or gradient descent for more complex problems, then this procedure continues post data where the data are now replaced by sample data from the model using the current parameter estimate. This is done in such a way so that the ensuing sequence of parameter estimators forms a martingale.
Here, it is worth pointing out about the form for , constructed sequentially. Unless the model is based on exchangeability, the ordering of the will matter. For example, without exchangeability, the joint density of will not share the same density as . We are not aware of any practical consequence of this. As well, if the sequence is c.i.d. the proximity to an exchangeable sequence depends on the sample size and asymptotically a c.i.d. is exchangeable. On the other hand, if exchangeability is regarded as an essential component to Bayesian uncertainty quantification, then the prior to posterior will be the best choice.
5. Illustrations
Example 5.1. —
The first and obvious example is the normal location parameter; the martingale here is for with . So which is based on the being the sample mean at each iteration. Here, the are independent standard normal random variables. The frequentist posterior is normal with mean , the sample mean from the observed data, and with variance . The sum is approximately .
If the variance is also unknown, we set up two martingales, one for the mean and one for the variance. Now we take the same martingale for the mean and take , where we take
The martingale for is explained as
which can be tidied up to give
Taking a conditional expectation keeping fixed,
which demonstrates the martingale.
We take a worked illustration with independent samples from a standard normal distribution. The value of we use to stop the martingales is 1000. This value appears large enough for the martingales to converge. A histogram of the 100 samples from the martingales we ran are presented in figure 1. The sample mean and sample variance from the data were 0.048 and 0.848, respectively.
Figure 1.
Marginal frequentist posterior distribution for and .
Example 5.2. —
Here, we consider the Poisson model; with density function
Hence, and . Taking , the martingale becomes
In this case, it is easy to compute the limit values of the mean and variance. So , due to the martingale, and
Note that the Bayesian objective posterior using the prior is gamma with mean and variance .
We ran an illustration using and taking independent samples from the Poisson distribution with mean 1. We collected 100 frequentist posterior samples by running 100 martingales each to a length of . The posterior is presented as the histogram and the objective Bayesian gamma posterior with prior is shown alongside in figure 2.
Figure 2.
Frequentist posterior (histogram) from the Poisson example, alongside the Bayesian gamma objective posterior (line).
Example 5.3. —
In this part, we consider a two parameter gamma model, . So and so
where is the digamma function, with
Defining the matrix and vector
where . Then the martingale is
We took samples from the gamma distribution with and and 500 martingales were run each with a run length of 1000. The frequentist distribution for is presented in figure 3.
Figure 3.
(a and b) Frequentist posteriors from the gamma example.
Example 5.4. —
This example looks at the standard linear regression model,
where is a vector of regression variables and is a vector of unknown coefficients. We assume the is known and the are independent standard normal errors. The estimator based on the observed sample of size is where is the matrix from .
We construct the martingale using
where . The martingale becomes, for ,
where . Here, the can be arbitrarily chosen or sampled from the observed values . The clear choice would be to sample from the empirical distribution.
For the specific illustration, we took samples with the true and all the sampled independently from a standard normal distribution, with . We ran 500 martingales and each was run for a length of size 1000. The 500 frequentist posterior samples are presented in figure 4.
Figure 4.
Frequentist posteriors from the linear model example.
Example 5.5. —
A further example is a Markov-driven time series with standard normal marginals. So
where the are independent standard normal random variables. Then . The log-density is given by
The first and second derivatives with respect to are easy to obtain, e.g.
Hence, the martingale is
where is normal with mean and variance .
By way of an illustration, we took and then the for , with , according to the Markov model with . The estimator turned out to be 0.758. We ran 100 martingales and each was run for a length of 500 iterations. The histogram of the 100 samples from the frequentist posterior is provided in figure 5.
Figure 5.
Frequentist posterior from the Markov time series model example.
6. Theory
The theory we present here is about martingales and the convergence of such to limits. For general information on martingales, see Williams [43]. The first result is for the sequence of random distributions , which form a martingale in the sense that
Since a probability distribution is bounded the converges almost surely to a for each . That the sequence converges with probability one to a random distribution function is a consequence of theorem 2.2 appearing in Berti et al. [29]. A modified version of the theorem relevant to the present paper is provided here.
Theorem 6.1. —
Given , if , is a sequence of random distribution functions, for , and converges almost surely to for each with , and is a tight distribution function, then converges weakly to the random distribution function almost surely.
From the construction of for , it is clearly evident that is a martingale for each . From the martingale convergence theorem, there exists a for each for which almost surely. Now is a fixed distribution function and hence for any there will be a compact set such that . Hence, as such, is a tight distribution function. For more on tightness, see Billingsley [44]. Hence, from the theorem, converges weakly to the random distribution function almost surely.
For a general martingale for the parameter , we have the following martingale convergence theorem:
Theorem 6.2. —
If is a martingale and then there exists a random variable such that a.s. and .
In practice, we consider .
One of the martingales on which we rely is the stochastic gradient descent, or ascent in our case, algorithm. If are i.i.d. from and
6.1 for some suitable sequence and starting point , where , then we require that as in the sense that . See, for example, Murphy [45].
The essence of the argument for convergence is as follows:
The last term on the right side is the important part and write the expectation as
where
and the assumption is that for sufficiently close to , it is that if , if and , and further that within a neighbourhood for some finite . Hence, we can write
Proceeding, for some constants and , we have
and recall is of order . It is now standard mathematics to demonstrate that .
This supports (6.1) as a Bayesian style learning algorithm, updating the Bayesian estimator with data. Once we reach the end of the data, we revert to a martingale, generating the data with the current point estimator and proceeding with (6.1) to infinity. The exists and has expected value and the variance, just as with usual Bayesian updating, is . Hence, the ‘posterior’ is available and which we regard as well motivated as the traditional Bayesian approach.
In order for the martingale to converge, we need to look at . Now if
| 6.2 |
then
where the variance applies to the given and the expectation with respect to . Hence,
which we need to ensure is finite. This can be achieved through a focus on , the interpretation of which is to be found in (6.2) and illustrated in the examples we have presented, specifically how it is related to posterior variances involving the Fisher information.
7. Discussion
From the understanding of Bayesian inference through the construction of in order to obtain we relax the strict assumptions which take this predictive to start with a prior distribution on a parameter. We argue it is straightforward to construct such a predictive, using density estimators, for example, even based on maximum-likelihood estimators, which in the non-parametric case could be represented by the empirical distribution function.
A key to the construction of is the martingale, indeed, it is a key component of the Bayesian framework, though often concealed from view as the prior to posterior update puts it firmly into the background. The martingale ensures convergence; i.e. the existence of what we have named the frequentist posterior. On the other hand, it is the step-by-step density estimators, or predictives, , which provide the accuracy.
After the data are exhausted, it is the which creates the uncertainty; rather than the notion of an alternative dataset of size that could have been seen in the Frequentist approach. Simulating this from the predictive to form generates random outcomes of parameter values using the estimate , and these form the basis of the frequentist posterior distribution. Each sampled represents a potentially true value and the collection quantifies the associated uncertainty.
For readers remaining unclear on the ideas, let us return to the standard Bayesian approach understood from the missing data perspective. Given the Bayesian derives what is regarded as the ‘best’ density estimator for via the posterior predictive (1.2). This is sampled, the model is updated using the sampled value and the procedure moves on to , and so on. In the limit, the object of interest, let us assume it is , constructed from , is a sample from the posterior based on the sample of size . The key is the martingale sequence of estimators of as the samples are obtained.
The extended view of this strategy promoted in the paper is what exactly is the best density estimator for given for any . Many density estimators are available. However, the Bayesian procedure tells us how to perform posterior analysis using any density estimator, including classical ones, such as those based on maximum-likelihood estimators.
Acknowledgements
The authors are grateful for the comments and suggestions of three reviewers which have allowed us to improve the presentation of the paper.
Data accessibility
This article has no additional data.
Authors' contributions
C.C.H.: conceptualization, formal analysis, investigation, methodology, writing–original draft, writing–review and editing; S.G.W.: conceptualization, formal analysis, investigation, methodology, writing–original draft, writing–review and editing.
Both authors gave final approval for publication and agreed to be held accountable for the work performed therein.
Conflict of interest declaration
We declare we have no competing interests.
Funding
No funding has been received for this article.
References
- 1.Bernardo JM, Smith AFM. 2009. Bayesian theory, vol. 405. Chichester, UK: John Wiley & Sons. [Google Scholar]
- 2.de Finetti B. 1937. La prevision: ses lois logiques, ses sources subjectives. Ann. Inst. H. Poincare 7, 1-68. [English translation in Studies in Subjective Probability (1980) (H. E. Kyburg and H. E. Smokler, eds.) 53-118. Krieger, Malabar, FL. [Google Scholar]
- 3.Hewitt E, Savage LJ. 1955. Symmetric measures on Cartesian products. Trans. Am. Math. Soc. 80, 470-501. ( 10.1090/S0002-9947-1955-0076206-8) [DOI] [Google Scholar]
- 4.Cifarelli DM, Regazzini E. 1996. De Finetti’s contribution to probability and statistics. Stat. Sci. 11, 253-282. ( 10.1214/ss/1032280303) [DOI] [Google Scholar]
- 5.Lindley DV, Smith AF. 1972. Bayes estimates for the linear model. J. R. Stat. Soc. B (Methodological), with discussion 34, 1-41. ( 10.1111/j.2517-6161.1972.tb00885.x) [DOI] [Google Scholar]
- 6.Roberts HV. 1965. Probabilistic prediction. J. Am. Stat. Assoc. 60, 50-62. ( 10.1080/01621459.1965.10480774) [DOI] [Google Scholar]
- 7.Ericson WA. 1969. Subjective Bayesian models in sampling finite populations. J. R. Stat. Soc. B (Statistical Methodology) 31, 195-224. ( 10.1111/j.2517-6161.1969.tb00782.x) [DOI] [Google Scholar]
- 8.Geisser S. 1982. Aspects of the predictive and estimative approaches in the determination of probabilities. Biometrics 38, 75-85. ( 10.2307/2529856) [DOI] [Google Scholar]
- 9.Geisser S. 1983. On the prediction of observables: a selective update. Technical report, University of Minnesota.
- 10.Lo AY. 1988. A Bayesian bootstrap for a finite population. Annals of Statistics 16, 1684-1695. [Google Scholar]
- 11.Ghosh M, Marsden G. 1997. Bayesian Methods for Finite Population Sampling. New York, USA: Routledge. [Google Scholar]
- 12.Fortini S, Ladelli L, Regazzini E. 2000. Exchangeability, predictive distributions and parametric models. Sankhya A Indian J. Stat. A. 62, 86-109. [Google Scholar]
- 13.Fortini S, Petrone S. 2012. Predictive construction of priors in Bayesian nonparametrics. Braz. J. Probab. Stat. 26, 423-449. ( 10.1214/11-BJPS176) [DOI] [Google Scholar]
- 14.Fortini S, Petrone S. 2014. Predictive distribution (de Finetti’s view). Wiley StatsRef: Statistics Reference Online, pp. 1–9.
- 15.Lijoi A, Pruenster I, Walker SG. 2007. Bayesian consistency for stationary models. Econometr. Theory 23, 749-759. ( 10.1017/S0266466607070314) [DOI] [Google Scholar]
- 16.Lijoi A, Mena RH, Pruenster I. 2007. Bayesian nonparametric estimation of the probability of discovering a new species. Biometrika 94, 769-786. ( 10.1093/biomet/asm061) [DOI] [Google Scholar]
- 17.Favaro S, Pruenster I. 2009. Bayesian nonparametric inference for species variety with a two parameter Poisson-Dirichlet process prior. J. R. Stat. Soc. B 71, 993-1008. ( 10.1111/j.1467-9868.2009.00717.x) [DOI] [Google Scholar]
- 18.De Blasi P, Favaro S, Lijoi A, Mena R, Pruenster I, Ruggiero M. 2015. Are Gibbs–type priors the most natural generalization of the Dirichlet process? IEEE Trans. Pattern Anal. Mach. Intell. 37, 212-229. ( 10.1109/TPAMI.2013.217) [DOI] [PubMed] [Google Scholar]
- 19.Kass RE, Raftery AE. 1995. Bayes factors. J. Am. Stat. Assoc. 90, 773-795. ( 10.1080/01621459.1995.10476572) [DOI] [Google Scholar]
- 20.Fong E, Holmes CC. 2020. On the marginal likelihood and cross-validation. Biometrika 107, 489-496. ( 10.1093/biomet/asz077) [DOI] [Google Scholar]
- 21.Gneiting T, Raftery AE. 2007. Strictly proper scoring rules, prediction, and estimation. J. Am. Stat. Assoc. 102, 359-378. [Google Scholar]
- 22.Maitra A. 1977. Integral representations of invariant measures. Trans. Am. Math. Soc. 229, 209-225. ( 10.1090/S0002-9947-1977-0442197-5) [DOI] [Google Scholar]
- 23.Aldous DJ 1985. Exchangeability and related topics. Lectures notes in Mathematics, 1117. New York, NY: Springer. [Google Scholar]
- 24.Berti P, Pratelli L, Rigo P. 2004. Limit theorems for a class of identically distributed random variables. Ann. Probab. 32, 2029-2052. ( 10.1214/009117904000000676) [DOI] [Google Scholar]
- 25.Robert CP. 2007. The Bayesian choice. New York, NY: Springer. [Google Scholar]
- 26.Camerlenghi F, Lijoi A, Pruenster I. 2017. Bayesian prediction with multiple–samples information. J. Multivariate Anal. 156, 18-28. ( 10.1016/j.jmva.2017.01.010) [DOI] [Google Scholar]
- 27.Fong E, Holmes C, Walker SG. 2023. Martingale posterior distributions. J. R. Stat. Soc. B. In press. [Google Scholar]
- 28.Doob JL. 1949. Application of the theory of martingales. Actes du Colloque International Le Calcul des Probabilites et ses applications (Lyon, 28 Juin–3 Juillet 1948), Paris CNRS, 23–27.
- 29.Berti P, Pratelli L, Rigo P. 2006. Almost sure weak convergence of random probabilty measures. Stoch. Int. J. Probab. Stoch. Process. 78, 91-97. ( 10.1080/17442500600745359) [DOI] [Google Scholar]
- 30.Hahn PR, Martin R, Walker SG. 2018. On recursive Bayesian predictive distributions. J. Am. Stat. Assoc. 113, 1085-1093. ( 10.1080/01621459.2017.1304219) [DOI] [Google Scholar]
- 31.Rubin DB. 1981. The Bayesian bootstrap. Ann. Stat. 9, 130-134. ( 10.1214/aos/1176345338) [DOI] [Google Scholar]
- 32.Efron B. 1979. Bootstrap methods: another look at the jackknife. Ann. Stat. 7, 1-26. ( 10.1214/aos/1176344552) [DOI] [Google Scholar]
- 33.Ferguson TS. 1973. A Bayesian analysis of some nonparametric problems. Ann. Stat. 1, 209-230. ( 10.1214/aos/1176342360) [DOI] [Google Scholar]
- 34.Blackwell D, MacQueen JB. 1973. Ferguson distributions via Polya urn schemes. Ann. Stat. 1, 353-355. ( 10.1214/aos/1176342372) [DOI] [Google Scholar]
- 35.Sethuraman J. 1984. A constructive definition of Dirichlet priors. Statistica Sinica 4, 639-650. [Google Scholar]
- 36.Fong E, Lyddon S, Holmes C. 2019. Scalable nonparametric sampling from multimodal posteriors with the posterior bootstrap. In Int. Conf. on Machine Learning (pp. 1952–1962). PMLR.
- 37.Lyddon S, Walker S, Holmes CC. 2018. Nonparametric learning from Bayesian models with randomized objective functions. Adv. Neural Inf. Process. Syst. 31, 2075-2085. [Google Scholar]
- 38.Lyddon SP, Holmes CC, Walker SG. 2019. General Bayesian updating and the loss-likelihood bootstrap. Biometrika 106, 465-478. ( 10.1093/biomet/asz006) [DOI] [Google Scholar]
- 39.Newton MA, Polson NG, Xu J. 2021. Weighted Bayesian bootstrap for scalable posterior distributions. Can. J. Stat. 49, 421-437. ( 10.1002/cjs.11570) [DOI] [Google Scholar]
- 40.Newton MA, Raftery AE. 1994. Approximate Bayesian inference with the weighted likelihood bootstrap. J. R. Stat. Soc. B (Methodological) 56, 3-26. ( 10.1111/j.2517-6161.1994.tb01956.x) [DOI] [Google Scholar]
- 41.Nie L, Ročková V. 2023. Bayesian bootstrap spike-and-slab LASSO. J. Am. Stat. Assoc. In press. ( 10.1080/01621459.2022.2025815) [DOI] [Google Scholar]
- 42.Efron B. 2012. Bayesian inference and the parametric bootstrap. Ann. Appl. Probab. 6, 1971-1997. ( 10.1214/12-AOAS571) [DOI] [PMC free article] [PubMed] [Google Scholar]
- 43.Williams D. 1991. Probability with martingales. Cambridge, UK: Cambridge University Press. [Google Scholar]
- 44.Billingsley P. 1999. Convergence of probability measures. New York, NY: John Wiley & Sons, Inc. [Google Scholar]
- 45.Murphy K. 2021. Probabilistic machine learning: an introduction. Cambridge, MA: MIT Press. [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Data Availability Statement
This article has no additional data.





