Double-Parallel Monte Carlo for Bayesian Analysis of Big Data

Abstract

This paper proposes a simple, practical and efficient MCMC algorithm for Bayesian analysis of big data. The proposed algorithm suggests to divide the big dataset into some smaller subsets and provides a simple method to aggregate the subset posteriors to approximate the full data posterior. To further speed up computation, the proposed algorithm employs the population stochastic approximation Monte Carlo (Pop-SAMC) algorithm, a parallel MCMC algorithm, to simulate from each subset posterior. Since this algorithm consists of two levels of parallel, data parallel and simulation parallel, it is coined as “Double Parallel Monte Carlo”. The validity of the proposed algorithm is justified mathematically and numerically.

1 Introduction

The MCMC method has proven to be a very powerful tool for analyzing data of complex structures. However, it is difficult to be applied to big data problems for which complex models are often needed. The difficulty comes from two aspects. The first one is on data storage; the dataset can be too large for a single computer to store and process. The second one is on computational time; the MCMC method can be very time consuming for simulating from the posterior of a large data set, which typically requires a large number of iterations and a complete scan of the full dataset for each iteration. However, thanks to the strategy of embarrassingly parallel computing, the two issues can now be solved simultaneously.

The strategy of embarrassingly parallel computing is to divide a large dataset into a number of smaller subsets such that each subset can be stored in a single machine, and then conduct the Bayesian analysis for each subset separately. Finally, the posterior samples generated for each subset are aggregated in some way such that correct inference can be made for the full data posterior. During the past few years, this strategy has been pursued by a few groups enthusiastically, and several algorithms have been developed to address the issue of subset posterior aggregation.

To be a little more detailed, suppose that a large dataset has been partitioned into k subsets, and N posterior samples have been generated for each subset. Let $\{{\mathit{\theta }}_1^{(i)},\dots {\mathit{\theta }}_N^{(i)}\}$ denote the posterior samples generated from subset i. Based on the Bernstein-von Mises theorem, which states that the posterior tends to a normal distribution centered around the true parameter value θ* as the number of observations grows, Scott et al. (2016) proposed to use the weighted average ${\sum }_{i=1}^k{w}_i{\mathit{\theta }}_j^{(i)}$ to approximate a full data posterior sample, where the weight w_i is the inverse of the covariance matrix of $\{{\mathit{\theta }}_1^{(i)},\dots {\mathit{\theta }}_N^{(i)}\}$. This algorithm is exact when the subset posterior is Gaussian. Based on the same theory, Neiswanger et al. (2014) proposed to fit the posterior samples generated for each subset by a Gaussian density, denoted the fitted density by p̂_i for i = 1, …, k, and then draw samples from the product density p̂₁ … p̂_k. As an extension of this approach, Neiswanger et al. (2014) proposed to estimate the subset posterior density using a Gaussian kernel density estimation method or a semiparametric density estimation method. Another method that makes use of kernel approximation is by Minsker et al. (2014), where the subset posteriors are combined by estimating a probability distribution that minimizes a loss function defined in the reproducing kernel Hilbert space embedding the subset posteriors. In addition, Wang and Dunson (2013) proposed a Weierstrass refinement sampler, where the samples from an initial approximation to the full data posterior (e.g., obtained via variational approximation or other methods) are refined using the information obtained from the subset posterior samples within a Weierstrass approximation. Later, Wang et al. (2015) proposed to combine the subsets using random partition trees. These methods generally work well, but their accuracy can vary significantly depending on how close the subset posteriors are to Gaussian, or the choice of kernel and its bandwidth, or the accuracy of density estimators for the subset posteriors, or whether the subset posteriors have non-overlapping supports. In particular, their accuracy can be low when the dimension of θ is high. Quite recently, the so-called WASP method was proposed by Srivastava et al. (2015a) and Srivastava et al. (2015b), where each subset posterior is approximated by an empirical measure and they are combined by estimating their barycenter in the Wasserstein space of probability measures. This method does not depend on the kernel density estimation any more, but computing the Wasserstein barycenter needs to solve a huge linear programming problem which often requires a lot of computer memory. To resolve the computational problem, Li et al. (2017) proposed a simple and general posterior interval estimation algorithm (PIE) by averaging quantiles estimated from each subset. However, this method was devised to estimate certain quantiles and can not be directly used to estimate the whole distribution.

In this paper, we propose a new method for aggregating subset posterior samples. The new method is surprisingly simple, which is to first simulate from some modified subset posteriors, for which the log-likelihood functions are appropriately scaled according to their sample size, and then recenter the subset posterior samples to their global mean. Under mild conditions, we show that the aggregated samples have the same convergence rate toward the true parameter θ* as those drawn from the full data posterior. The numerical results indicate that the new method can be rather accurate compared to the existing ones. In order to further speed up computation, we suggest to use the Pop-SAMC algorithm(Song et al., 2014), rather than traditional single chain MCMC algorithms, to draw samples from each subset posterior. Since the proposed method consists of two levels of parallel, data parallel and simulation parallel, it is coined as “Double Parallel” Monte Carlo.

The remainder of this article is organized as follows. Section 2 presents the proposed sample aggregation method and describes its theoretical properties. Section 3 first gives a brief review of the pop-SAMC algorithm, and then discusses the Double Parallel strategy. Sections 4 and 5 present some numerical results along with comparisons with some existing methods. Section 6 concludes the article with a brief discussion.

2 Subset Posterior Aggregation

Suppose that a random sample X = {X₁, …, X_n} has been collected from the distribution f(x|θ*), where θ* ∈ Θ ⊂ ℝ^p and Θ is the parameter space. Let g(θ) denote the prior distribution of θ. Then the posterior distribution of θ is given by

$$\pi (\mathit{\theta }|\mathit{X})=\frac{{\prod }_{i=1}^{n}f(X_i|\mathit{\theta })g(\mathit{\theta })}{{\int }_{\mathrm{\Theta }}{\prod }_{i=1}^{n}f(X_i|\mathit{\theta })g(\mathit{\theta })d\mathit{\theta }}.$$ (1)

In most cases, π(θ|X) is analytically intractable and we have to approximate it using the Markov chain Monte Carlo(MCMC) method. However, as mentioned previously, when n is very large, the MCMC method is computationally prohibitive because it requires a large number of scans of the dataset.

To address this issue, we divide the data into k subsets, each containing about the same number of samples. Let X_[j] = (X_j1,…, X_jmj) denote the jth subset, where m_j denotes the sample size of X_[j]. Let π(θ|X_[j]) denote the posterior distribution corresponding to the subset X_[j], for which the variance is approximately n/m_j times the variance of the full data posterior π(θ|X). To adjust the variance, for each subset, we instead work on a modified subset posterior

$$\stackrel{\sim }{\pi }(\mathit{\theta }|{\mathit{X}}_{[j]})=\frac{{\prod }_{i=1}^{m_j}{f}^{n/m_j}(X_{\mathit{\text{ji}}}|\mathit{\theta })g(\mathit{\theta })}{{\int }_{\mathrm{\Theta }}{\prod }_{i=1}^{m_j}{f}^{n/m_j}(X_{\mathit{\text{ji}}}|\mathit{\theta })g(\mathit{\theta })d\mathit{\theta }},$$ (2)

where each sample is duplicated n/m_j times. Such a modification, first introduced in Minsker et al. (2014), ensures that π̃(θ|X_[j]) has about the same variance as the full data posterior. In what follows, we refer to π̃(θ|X_[j]) as a subposterior and, without loss of generality, assume that m₁ = m₂ = ⋯ = m_k = m holds.

Let μ⁽¹⁾,⋯, μ^(k) denote the mean vectors of the respective subposteriors, and let $\hat{\mathit{\mu }}=\frac{1}{k}{\sum }_{j=1}^k{\mathit{\mu }}^{(j)}$ denote their averages. We propose to recenter each of the subposteriors to μ̂ and then use the following mixture of re-centered subposteriors to approximate the full data posterior π(θ|X):

$$\stackrel{\sim }{\pi }(\mathit{\theta }|\mathit{X})=\frac{1}{k}\sum _{j=1}^k\stackrel{\sim }{\pi }(\mathit{\theta }-\hat{\mathit{\mu }}+{\mathit{\mu }}^{(j)}|{\mathit{X}}_{[j]}).$$ (3)

To quantify the accuracy of the approximation, we make the following assumptions:

• A1
  The log-likelihood function ${\sum }_{i=1}^n\text{log }f(X_i|\mathit{\theta })=-{\mathit{\text{nu}}}_n(\mathit{\theta })$ is six-times continuously differentiable on Θ and has local minima {θ̂_n : n = 1, 2, …}. Let B_δ(θ) denote an open ball of radius δ centered at θ, and let D²u_n(θ) denote the Hessian of u_n at θ. Further suppose there exist positive numbers ε, M, η and an integer n₀ such that when n ≥ n₀,

  1. For all θ ∈ B_ε(θ̂_n) and all 1 ≤ j₁,…, j_d ≤ p with 0 ≤ d ≤ 6, |∂_j1,…,jd u_n(θ)| < M.
  2. det(D²u_n(θ̂_n)) > η.
  3. for all δ for which 0 < δ < ε, B_δ (θ̂_n) ⊆ Θ and

     $$\text{lim}\underset{n\to \infty }{\text{sup}}\underset{\mathit{\theta }\in \mathrm{\Theta }\backslash B_{\delta }({\hat{\mathit{\theta }}}_n)}{\text{sup}}\{u_n({\hat{\mathit{\theta }}}_n)-u_n(\mathit{\theta })\}<0.$$
• A2θ* is an interior point of Θ, g(θ*) > 0, and g(θ) is four times continuous differentiable on Θ. In addition, ${\int }_{\mathrm{\Theta }}g(\mathit{\theta }){\Vert \mathit{\theta }\Vert }_2^2\text{ exp}\{-{\mathit{\text{nu}}}_n(\mathit{\theta })\}d\mathit{\theta }<\infty$ for n = 1, 2,…
• A3The number of subsets k can increase slowly with n, but should be at the rate of o(n^1/2).

Since the quantification involves posterior expansions based on Laplace’s method, the Laplace regularity condition (A1) is assumed. Refer to Kass et al. (1990) for more details. This condition is standard and generally holds for the exponentially family. Under the above conditions, we have the following theorem, whose proof is given in the appendix.

Theorem 1

If the conditions A1–A3 are satisfied, then we have

$$E_{\stackrel{\sim }{\pi }}(\mathit{\theta })-E_{\pi }(\mathit{\theta })=O_p(m^{-1}),$$ (4)

$${\mathit{\text{Var}}}_{\stackrel{\sim }{\pi }}(\mathit{\theta })-{\mathit{\text{Var}}}_{\pi }(\mathit{\theta })=o_p(n^{-1}),$$ (5)

$$d^2(\stackrel{\sim }{\pi }(\mathit{\theta }|\mathit{X}),{\delta }_{{\mathit{\theta }}^{\ast }})-d^2(\pi (\mathit{\theta }|\mathit{X}),{\delta }_{{\mathit{\theta }}^{\ast }})=o_p(n^{-1}),$$ (6)

where E_π and E_π̃ denote the expectations with respect to π(θ|X) and π̃(θ|X), respectively; Var_π and Var_π̃ denote the variances with respect to π(θ|X) and π̃(θ|X), respectively; and $d^2(\stackrel{\sim }{\pi }(\mathit{\theta }|\mathit{X}),{\delta }_{{\mathit{\theta }}^{\ast }})={\int }_{\mathrm{\Theta }}{\Vert \mathit{\theta }-{\mathit{\theta }}^{\ast }\Vert }_2^2\stackrel{\sim }{\pi }(\mathit{\theta }|\mathit{X})d\mathit{\theta }$ is the square of the Wasserstein distance of order 2 between π̃(θ|X) and the Dirac measure at θ*.

Equations (4) and (5) measure the accuracy of the approximation of π̃(θ|X) to π(θ|X) in terms of mean and variance, respectively. Usually, d²(π(θ|X), δ_θ*) is O(n⁻¹), thus equation (6) implies that π̃(θ|X) and π(θ|X) share the same convergence rate toward the true value θ*. In other words, the subposterior aggregation does not lose much information about the data.

Rather than θ itself, sometimes we are interested in h(θ), a ℝ^p ↦ ℝ^q function of θ. A similar result, which measures the accuracy of the approximation π̃(h(θ)|X), can be obtained if we further have the following assumption:

• A4h(θ) is four times continuously differentiable on Θ and satisfies ${\int }_{\mathrm{\Theta }}g(\mathit{\theta }){\Vert h(\mathit{\theta })\Vert }_2^2\text{ exp}\{-{\mathit{\text{nu}}}_n(\mathit{\theta })\}d\mathit{\theta }<\infty$ for n = 1, 2, …

Corollary 1

If the conditions A1–A4 are satisfied, then we have

$$E_{\stackrel{\sim }{\pi }}h(\mathit{\theta })-E_{\pi }h(\mathit{\theta })=O_p(m^{-1}),$$

$${\mathit{\text{Var}}}_{\stackrel{\sim }{\pi }}h(\mathit{\theta })-{\mathit{\text{Var}}}_{\pi }h(\mathit{\theta })=o_p(n^{-1}),$$

$$d^2(\stackrel{\sim }{\pi }(h(\mathit{\theta })|\mathit{X}),{\delta }_{h({\mathit{\theta }}^{\ast })})-d^2(\pi (h(\mathit{\theta })|\mathit{X}),{\delta }_{h({\mathit{\theta }}^{\ast })})=o_p(n^{-1}).$$

The proof is similar to that of Theorem 1, which is based on the expansion for the posterior mean of h(θ) and thus omitted in the paper.

3 Double Parallel Monte Carlo

In this section, we first give a brief review of the Pop-SAMC algorithm and discuss its implementation on the OpenMP platform. Then we describe the Double Parallel Monte Carlo scheme.

3.1 Pop-SAMC algorithm and its OpenMP implementation

As aforementioned, although MCMC is powerful for analyzing the data of complex structures, its computer-intensive nature precludes its use for big data analysis. To accelerate computation, one feasible way is to conduct parallel MCMC simulations. People have debated for a long time to make a single long run or many short runs. For conventional MCMC algorithms, such as the Metropolis-Hastings algorithm (Metropolis et al., 1953; Hasting, 1970) and the Gibbs sampler (Geman and Geman, 1984), parallel runs may not provide any theoretical advantages over a single long run. In general, if one cannot get a good answer with a long run, then one cannot get a good answer with many short runs either. However, this situation differs for the population stochastic approximation Monte Carlo (pop-SAMC) algorithm (Song et al., 2014), where it is shown that running pop-SAMC with κ chains (in parallel) for T iterations is asymptotically more efficient than running a single SAMC chain for κT iterations when the gain factor sequence decreases slower than O(1/t), where t indexes iterations. This is due to that the chains in pop-SAMC interact with each other intrinsically.

The pop-SAMC algorithm consists of two steps, population sampling and ξ-updating, where ξ denotes a vector of adaptive parameters evolving with iterations. In the population sampling step, each chain is updated independently for one or a few iterations. In the ξ-updating step, ξ_t (i.e., the value of ξ at iteration t) is updated based on the collected information from individual chains, which enforces interactions between different chains and, consequently, improves the efficiency of the algorithm. To make the paper self-contained, the algorithm is briefly described as follows.

Suppose that we are interested in simulating samples from a density function p(θ), θ ∈ Θ, and Θ has been partitioned into M subregions: E₁ = {θ : U(θ) < u₁}, E₂ = {θ : u₁ ≤ U(θ) < u₂}, …, E_M−1 = {θ : u_M−2 ≤ U(θ) < u_M−1}, and E_M = {θ : U(θ) ≥ u_M−1}, where U(θ) is a pre-specified function of θ, e.g., U(θ) = −log p(θ), and u₁ < u₂ < ⋯ < u_M−1 are pre-specified numbers. To explain the concept of SAMC, we assume for the time being that all the subregions are non-empty; that is, z_i = ∫_Ei p(θ)dθ > 0 for all i = 1, …, M. However, as explained in Liang et al. (2007), the algorithm does allow the existence of empty subregions. Let π = (π₁, …, π_M) denote the desired sampling distribution of the M subregions, where ${\sum }_{i=1}^M{\pi }_i=1$ and π_i > 0 for all i = 1, …, M. Given the partition and the desired sampling distribution, Pop-SAMC seeks to draw samples from the distribution

$$p_{\mathit{z}}(\mathit{\theta })\propto \sum _{i=1}^M\frac{{\pi }_{i}p(\mathit{\theta })}{z_i}I(\mathit{\theta }\in E_i).$$

If z_i’s are known and the space is partitioned appropriately, e.g., the energy bandwidth of each subregion is small enough, then the sampling will lead to a random walk in the space of subregions and thus the local-trap problem can be overcome essentially. However, since z₁, …, z_M are generally unknown, Pop-SAMC employs the stochastic approximation algorithm (Robbins and Monro, 1951) to learn their values (up to a constant factor) in an adaptive way.

Let κ denote the population size, i.e., the number of parallel Markov chains contained in Pop-SAMC, and let θ_t = (θ_t1, …, θ_tκ) denote the current state of the κ chains. Let ξ_t = (ξ_t1, …, ξ_tM) denote the working estimate of (log(z₁/π₁), …, log(z_M/π_M)) obtained at iteration t. One iteration of the algorithm consists of the following steps:

1. (Population sampling) For i = 1, …, κ, generate a new sample θ_t,i starting from θ_t−1,i by a single MH update with the target distribution given by

   $$p_{{\mathit{\xi }}_{t-1}}(\mathit{\theta })\propto \sum _{j=1}^M\frac{p(\mathit{\theta })}{e^{{\xi }_{t-1,j}}}I(\mathit{\theta }\in E_j).$$ (7)

2. (ξ-update) Set ξ_t = ξ_t−1 + γ_t(H_t − (1/M)1), where ${\mathit{H}}_t={({\sum }_{i=1}^{\kappa }I({\mathit{\theta }}_{t,i}\in E_1)/\kappa ,\cdots ,{\sum }_{i=1}^{N}I({\mathit{\theta }}_{t,i}\in E_M)/\kappa )}^T$, and γ_t is a gain factor.

To ensure the convergence of the algorithm, the gain factor {γ_t} is required to satisfy the conditions:

$$\sum _{t=1}^{\infty }{\gamma }_t=\infty ,\frac{{\gamma }_{t+1}-{\gamma }_t}{{\gamma }_t}=O({\gamma }_{t+1}^{\tau }),\sum _{t=1}^{\infty }\frac{{\gamma }_t^{(1+\tau \prime )/2}}{\sqrt{t}}<\infty ,$$

for some τ ∈ [1, 2) and τ′ ∈ (0, 1). For example, one can set γ_t = O(1/t^ζ) for ζ ∈ (1/2, 1]. To accommodate the case that ξ_t takes values in an unbounded space, a varying truncation version of the algorithm can be considered as in Andrieu et al. (2005).

Like the SAMC algorithm(Liang et al., 2007), Pop-SAMC possesses the self-adjusting mechanism, which operates based on past samples and enables the simulation to be immune to local traps. This can be considered as a significant advantage over conventional MCMC algorithms, such as the Metropolis-Hastings algorithm and the Gibbs sampler. Also, we would like to state that the pop-SAMC algorithm is essentially a dynamic importance sampling algorithm for which the trial distribution, i.e., the working target distribution (7), changes from iteration to iteration, and the quantities of interest can be estimated through weighted averaging as in conventional importance sampling (Liang, 2009). That is, Pop-SAMC generates a sequence of importance samples {(θ_t,1, e^{ξ_t,J(θt,1)}), …, (θ_t,κ, e^{ξ_t,J(θt,κ)})}, where J(θ_t,i) denotes the index of the subregion that θ_t,i belongs to, and e^{ξ_t,J(θt,i)} Specifies the importance weight of θ_t,i.

OpenMP is an application programming interface (API) for parallel programming on multi-core CPUs which are now available in regular desktops/laptops. It works in a shared memory mode with the fork/join parallelism, and is particularly suitable for pop-SAMC. To be precise, the population sampling step of pop-SAMC can be carried out in parallel through the pragma omp parallel to fork multiple threads with each thread running for an individual Markov chain. After the parallel execution, the threads join back to the master thread, where ξ_t is updated based on the information collected from the multiple threads. Since OpenMP works in a shared memory mode, distributing the updated ξ_t to different threads is avoided. Since the population sampling steps cost the major portion of the CPU, the parallel execution provides a nearly linear speedup for the simulation.

3.2 Double Parallel Monte Carlo

Based on the subposterior aggregation theory studied in Section 2 and the Pop-SAMC algorithm, we propose the following Double Parallel Monte Carlo algorithm for Bayesian analysis of big data.

• (Data Parallel) Divide the dataset into k subsets with each being about of the same sample size.

• (Simulation Parallel) Run Pop-SAMC for each subposterior π̃(θ|X_[i]) separately. Let $\{({\mathit{\theta }}_1^{(i)},w_1^{(i)}),\dots ,({\mathit{\theta }}_N^{(i)},w_N^{(i)})\}$ denote the importance samples generated by Pop-SAMC from π̃(θ|X_[i]) for i = 1, …, k. Let ${\hat{\mathit{\mu }}}^{(i)}={\sum }_{j=1}^N{w}_j^{(i)}{\mathit{\theta }}_j^{(i)}/{\sum }_{j=1}^N{w}_j^{(i)}$ denote the mean of the subposterior π̃(θ|X_[i]).

• (Sample aggregation) Calculate the global mean $\hat{\mathit{\mu }}={\sum }_{i=1}^k{\hat{\mathit{\mu }}}^{(i)}/k$, recenter the importance samples as $\{({\mathit{\theta }}_1^{(i)}-{\hat{\mathit{\mu }}}^{(i)}+\hat{\mathit{\mu }},w_1^{(i)}),\dots ,({\mathit{\theta }}_N^{(i)}-{\hat{\mathit{\mu }}}^{(i)}+\hat{\mathit{\mu }},w_N^{(i)})\}$ for i = 1, …, k.

Then, for each i = 1, 2, …, k, the re-centered samples can be viewed as a batch of importance samples generated from the full data posterior. For any function h(θ) that satisfies A4, the expectation ρ = E_πh(θ) can be naturally estimated by ${\hat{\rho }}_1={\sum }_{i=1}^k{\hat{\rho }}_1^{(i)}/k$, where ${\hat{\rho }}_1^{(i)}={\sum }_{j=1}^N{w}_j^{(i)}h({\mathit{\theta }}_j^{(i)}-{\hat{\mathit{\mu }}}^{(i)}+\hat{\mathit{\mu }})/{\sum }_{j=1}^N{w}_j^{(i)}$. Alternatively, ρ can be estimated by

$${\hat{\rho }}_2=\frac{{\sum }_{i=1}^k{\sum }_{j=1}^N{w}_j^{(i)}h({\mathit{\theta }}_j^{(i)}-{\hat{\mathit{\mu }}}^{(i)}+\hat{\mathit{\mu }})}{{\sum }_{i=1}^k{\sum }_{j=1}^N{w}_j^{(i)}}.$$

Let $U_i={\sum }_{j=1}^N{w}_j^{(i)}h({\mathit{\theta }}_j^{(i)}-{\hat{\mathit{\mu }}}^{(i)}+\hat{\mathit{\mu }}),S_i={\sum }_{j=1}^N{w}_j^{(i)}$, S = E(S_i) and V_i = U_i − ρS_i. Following from the property of SAMC, the variances of U_i and V_i are both finite. Then the standard error of ρ̂₂ can be calculated as for the ratio estimate (Ripley, 1987). The V_i’s can be treated as iid random variables with zero mean and finite variance, and its variance can be estimated by ${\hat{\sigma }}_V^2=1/k{\sum }_{i=1}^k{V}_i^2$. The law of large numbers implies that $1/\sqrt{k}{\sum }_{i=1}^k{V}_i$ is asymptotically normal $N(0,{\sigma }_V^2)$ and that

$$\sqrt{k}({\hat{\rho }}_2-\rho )=\frac{\frac{1}{\sqrt{k}}{\sum }_{i=1}^k{V}_i}{\frac{1}{k}{\sum }_{i=1}^k{S}_i}\to N(0,{\sigma }^2),$$

where ${\sigma }^2={\sigma }_V^2/S^2$, and it can be estimated by ${\hat{\sigma }}_V^2/{\hat{S}}^2$ with $\hat{S}={\sum }_{i=1}^k{S}_i/k$.

4 Simulation Study

4.1 Logistic Regression

The first example is very simple, whose goal is to show the validity of the proposed subposterior aggregation method. The example is adopted from Srivastava et al. (2015b). It is for a logistic regression with n = 10⁴ and the true parameter θ* = (1, −1)^T. The covariates Z₁ and Z₂ are drawn from the standard Gaussian distribution. The prior distribution of θ is N(0, I₂). To follow the notation in Section 2, we let X = (Y, Z₁, Z₂).

To implement the proposed Double Parallel algorithm, we randomly divided the dataset into 10 subsets with each consisting of 1000 samples. Then Pop-SAMC was run for each subset. Specifically, for each subset, we partitioned the parameter space Θ according to the energy function U(θ) = −log p_j(θ) with an equal bandwidth Δu = 0.5 into five subregions E₁ = {θ : U(θ) < u+0.5}, E₂ = {θ : u+0.5 ≤ U(θ) < u+1}, E₃ = {θ : u + 1 ≤ U(θ) < u + 1.5} E₄ = {θ : u + 1.5 ≤ U(θ) < u + 2}, and E₅ = {θ : U(θ) ≥ u + 2}, where p_j denote the subposterior of the jth subset, and u was chosen as the smallest value of U(θ) obtained in a preliminary trial. The gain factor γ_t was set as 100/ max(100, t). The proposal was set as a Gaussian random walk distribution with the covariance matrix 0.2²I₂. The population size was set to N = 10 and the number of iterations was set to T = 10⁵. The first 10⁴ iterations were discarded for the burn-in process, and samples were collected from the remainder of the run at every 5 iterations. In total, we had 1.8 × 10⁵ importance samples collected at the end of each run.

Figure 1 shows the contour plots of the full data posterior π(θ|X), each subposterior π̃(θ|X_[j]), and the proposed mixture posterior π̃(θ|X). The R package KernSmooth (Wand, 2015) was used to generate the corresponding binned kernel density estimates. The plots indicate that each subposterior has a similar shape with the full data posterior, however, most of them have a notably biased center from the true parameter θ*. By shifting the mean of each subposterior to the global mean, the bias was successfully removed. The mixture posterior π̃(θ|X) closely matches the full data posterior π(θ|X).

Figure 1.

Binned kernel posterior density estimates for the parameters of a logistic regression. The true parameter θ* = (1,−1)^T (black dot).

4.2 Nonlinear Regression

We use the following nonlinear regression example to compare the approximation accuracy of the proposed Double Parallel algorithm, posterior interval estimation(PIE) (Li et al., 2017) and Consensus Monte Carlo (Scott et al., 2016):

$$y_i={\theta }_1\sqrt{\text{exp}({\theta }_2{z}_{i1})}+{\theta }_3\sqrt{\text{exp}({\theta }_4{z}_{i2}+{\theta }_5{z}_{i3})}+{\theta }_6{z}_{i4}+{\theta }_7\text{sin}({\theta }_8{z}_{i5}+\pi /6)+{\theta }_9\frac{\text{exp}({\theta }_{10}{z}_{i6})}{1+\text{exp}({\theta }_{10}{z}_{i6})}+{\epsilon }_i,$$

for i = 1, 2, …, n, where θ = (θ₁, θ₂, θ₃, θ₄, θ₅, θ₆, θ₇, θ₈, θ₉, θ₁₀) = (2, 2, 2, 2, −2, −2, 5, 2, 4, 2) are the true parameters, and ε₁, ⋯, ε_n are i.i.d. normal random errors with mean 0 and variance σ² = 0.25. The covariates z₁ and z₂ are drawn from the standard normal distribution independently. Set the covariates z₃ = 0.7z₂+0.3e and z₄ = 0.7z₁+0.3e′, where e and e′ are also drawn from the standard normal distribution. Under this setting, z₂ and z₃ are highly correlated with a correlation coefficient of 0.919, so do z₁ and z₄. The covariates z₅ and z₆ are drawn independently from a t-distribution with the degree of freedom 10. We generated n = 10⁴ samples from this model and used the non-informative prior g(θ) ∝ 1 for all the parameters. To follow the notation in Section 2, we set X_i = (y_i, z_i1, z_i2, z_i3, z_i4, z_i5, z_i6).

For the Double Parallel algorithm, we randomly divided the dataset into 10 subsets with each consisting of 1000 samples. Pop-SAMC was run for each subset separately with the same setting as for the previous example except that the energy bandwidth was set to Δu = 2 and the covariance matrix of the Gaussian random walk proposal distribution was set to 0.01²I₁₀. For comparison, Consensus Monte Carlo and PIE were also applied to this example. Note for Consensus Monte Carlo, the subset posterior is defined as

$$\frac{{\prod }_{i=1}^{m}f(X_{\mathit{\text{ji}}}|\mathit{\theta })g^{1/k}(\mathit{\theta })}{{\int }_{\mathrm{\Theta }}{\prod }_{i=1}^{m}f(X_{\mathit{\text{ji}}}|\mathit{\theta })g^{1/k}(\mathit{\theta })d\mathit{\theta }},$$ (8)

which is slightly different from the subposterior defined in (2), the one used in PIE and Double Parallel.

To evaluate the approximation accuracy of Double Parallel, PIE and Consensus Monte Carlo, we utilized two criteria. The first criterion is the credible interval. The similarity between the credible intervals constructed by a method and those constructed by the full data-based MCMC method indicates the approximation accuracy of the method. We measured the similarity by the p-value of the paired-t test for the length of credible intervals of different parameters. The second criterion is more direct, which uses the following function

$$\text{Accuracy}({\pi }_1,{\pi }_2)=1-\frac{1}{2}\int |{\pi }_1(\theta )-{\pi }_2(\theta )|d\theta ,$$ (9)

to measure the similarity between two density functions π₁ and π₂. This function lies in [0, 1], with a larger value indicating higher similarity. In our case, π₁ is the posterior density obtained by the method under evaluation, and π₂ is the posterior density generated based on the full data. These densities can be estimated based on the posterior samples using the bkde function in the R package KernSmooth (Wand, 2015).

Table 1 and Table 2 show, respectively, the 90% and 80% credible intervals produced by different methods. As indicated by the p-values, Double Parallel and PIE produced very similar credible intervals to the full data-based MCMC method, while Consensus Monte Carlo did not. Hence, in this criterion, we conclude that Double Parallel and PIE outperforms Consensus Monte Carlo for this example.

Table 1.

90% credible intervals for each parameter produced by full data MCMC, Double Parallel, PIE and Consensus Monte Carlo, where the p-value is calculated by a paired-t test for the length of credible intervals in comparison with the full data MCMC method.

	MCMC	Double Parallel	PIE	Consensus Monte Carlo
θ1	(1.965,2.016)	(1.966,2.014)	(1.966,2.014)	(1.958,2.018)
θ2	(1.995,2.012)	(1.996,2.012)	(1.995,2.012)	(1.995,2.014)
θ3	(1.968,2.040)	(1.967,2.038)	(1.967,2.038)	(1.970,2.055)
θ4	(1.962,2.024)	(1.964,2.024)	(1.964,2.025)	(1.947,2.021)
θ5	(−2.017,−1.950)	(−2.017,−1.951)	(−2.017,−1.950)	(−2.014,−1.932)
θ6	(−2.024,−1.979)	(−2.022,−1.978)	(−2.022,−1.978)	(−2.023,−1.971)
θ7	(4.984,5.007)	(4.984,5.007)	(4.984,5.007)	(4.983,5.007)
θ8	(1.997,2.001)	(1.997,2.001)	(1.997,2.001)	(1.997,2.001)
θ9	(3.972,4.060)	(3.972,4.066)	(3.972,4.066)	(3.956,4.051)
θ10	(1.944,2.033)	(1.944,2.039)	(1.944,2.038)	(1.954,2.046)
p-value	—	0.769	0.586	2.68 × 10−3

Table 2.

80% credible intervals for each parameter produced by full data-based MCMC, Double Parallel, PIE and Consensus Monte Carlo, where the p-value is calculated by a paired-t test for the length of credible intervals in comparison with the full data MCMC method.

	MCMC	Double Parallel	PIE	Consensus Monte Carlo
θ1	(1.972,2.012)	(1.971,2.008)	(1.971,2.008)	(1.968,2.011)
θ2	(1.996,2.009)	(1.998,2.011)	(1.998,2.011)	(1.997,2.011)
θ3	(1.975,2.031)	(1.976,2.030)	(1.976,2.030)	(1.978,2.044)
θ4	(1.969,2.018)	(1.970,2.017)	(1.971,2.017)	(1.958,2.015)
θ5	(−2.011,−1.957)	(−2.010,−1.958)	(−2.010,−1.958)	(−2.006,−1.945)
θ6	(−2.019,−1.984)	(−2.017,−1.982)	(−2.017,−1.982)	(−2.018,−1.979)
θ7	(4.986,5.004)	(4.987,5.005)	(4.987,5.005)	(4.986,5.004)
θ8	(1.997,2.001)	(1.997,2.001)	(1.997,2.001)	(1.997,2.001)
θ9	(3.981,4.050)	(3.982,4.056)	(3.982,4.056)	(3.969,4.041)
θ10	(1.953,2.023)	(1.954,2.027)	(1.954,2.027)	(1.963,2.032)
p-value	—	0.901	0.811	0.016

Table 3 shows the approximation accuracy of Double Parallel and Consensus Monte Carlo to the full data posterior distribution, for each of the ten parameters. We don’t include PIE here because (i) from previous results, we expect the approximation accuracy of PIE should be very similar to that of Double Parallel, and (ii) PIE is proposed for quantile approximation and cannot be directly used for density approximation. Table 3 shows that for some parameters, Consensus Monte Carlo has better performance, while for others, Double Parallel wins. We suspect that this may depend on the similarity of each parameter’s posterior distribution with the normal distribution. In the average of approximation accuracy of the ten parameters, we find that Double Parallel is still slightly better than Consensus Monte Carlo for this example: Double Parallel produced an average of 0.924 with a standard deviation of 0.012, while Consensus Monte Carlo produced an average of 0.902 with a standard deviation of 0.015.

Table 3.

Accuracy of approximation to the full data-based posterior, measured in (9), by Double Parallel (DP) and Consensus Monte Carlo (CMC), for each parameter;

	θ1	θ2	θ3	θ4	θ5	θ6	θ7	θ8	θ9	θ10
Double Parallel	0.875	0.867	0.899	0.915	0.929	0.901	0.965	0.972	0.946	0.966
Consensus Monte Carlo	0.903	0.913	0.854	0.850	0.853	0.915	0.978	0.973	0.868	0.912

5 A Big Data Example

The goal of this example is to show how efficient the Double Parallel algorithm can be compared to the traditional single chain MCMC algorithm for a big data problem. For this purpose, we applied the Double Parallel algorithm to the MiniBooNE particle identification dataset, which is available at the UCI machine learning repository. This dataset records 130,064 events (observations), including 36,499 signal events and 93,565 background events. Each observation consists of the event type (signal event or background event) and 50 associated particle variables. The task of the problem is to explore the relationship between the event type and the associated particle variables. The more detailed description for the dataset and its physical background can be found in Roe et al. (2005).

The problem can be naturally treated using a logistic regression, where the event type is used as the response variable and the 50 associated particle variables are used as the predictors. To identify the important variables that are associated with the event type, we let the regression coefficients be subject to a heavy-tail distribution, t(3), which belongs to the class of local shrinkage priors but is more moderate in shrinking large regression coefficients than the Lasso prior(Tibshirani, 1996). Other local shrinkage priors, such as the horseshoe prior(Carvalho et al., 2010), can also be used here without affecting on the efficiency of the proposed algorithm. In data preprocessing, we first removed 468 samples with missing observations and then randomly divide the remaining samples into 10 subsets of nearly the same sample size. For each subset, Pop-SAMC was run with the population size κ = 20. The sample space was partitioned with an energy bandwidth Δu = 1 and the subregions determined through a preliminary run. The gain factor was set as γ_t = min(1, (t/1000)^−0.6). The algorithm was run for 1.1 × 10⁵ iterations, where the first 10⁴ iterations were for the burn-in process and the samples collected from the remaining 10⁵ iterations were used for inference. At each iteration, 20 parameters was randomly selected to be updated along a random direction with a step size of 0.1. The acceptance rate was around 0.16, which indicates the effectiveness of the simulation. On a high-end Dell Precision T7610 Workstation with 24 cores, one run of Pop-SAMC costs about 9 minutes (wall clock time) or 166 minutes total CPU time.

For comparison, the single chain Metropolis-Hastings (MH) algorithm was directly applied to simulate from the full data posterior. The algorithm was run for 2.2×10⁶ (= 20×1.1×10⁵) iterations correspondingly, where the first 2.0×10⁵ were discarded for the burn-in process and the samples collected from the remaining iterations were used for inference. The MH algorithm used the same proposal as the Pop-SAMC and the resulting acceptance rate was also about 0.16. On the same computer, one run of the MH algorithm costs about 1,373 minutes (wall clock time) or 1,371 minutes CPU time. In wall clock time, the computational cost by the Double Parallel algorithm is only 0.67% of that by the single chain MH algorithm! We have also implemented Consensus Monte Carlo on this dataset. For each subset, we utilized MH algorithm to generate samples and for each subset, the MH algorithm was run for 2.2 × 10⁶ iterations. To be fair to Consensus Monte Carlo, we only reported its results, but not its computational time, since Consensus Monte Carlo can also be implemented by Pop-SAMC, and in this case, it should use similar time with our method.

Table 4 shows the computational results produced by the three methods. For each method, we reported only the ten most significant variables, including their posterior mean and standard deviation. Here the significance of each variable was measured according to the ratio of its posterior mean and standard deviation. The results from the Double Parallel algorithm is surprisingly consistent with the result from the MH algorithm: All the variables have about the same posterior mean and standard deviation. The top 10 significant variables are exactly the same, even in the same order! This again indicates the validity of the Double Parallel algorithm for approximation of the full data posterior. Consensus Monte Carlo also selected the same top 10 significant variables, but in a slightly different order.

Table 4.

Comparison of computational time (wall clock time) and parameter estimation for the MiniBooNE particle data set

MH Algorithm		Double Parallel(k = 10, κ = 20)		Consensus Monte Carlo
wall clock time 1,373 minutes		9.2 minutes		–
Top 10 significant variables
var13	−1.2140 (0.0158)	var13	−1.2273 (0.0153)	var13	−1.2349(0.0156)
var1	−1.3529 (0.0328)	var1	−1.3667 (0.0338)	var1	−1.3804(0.0325)
var16	3.3730 (0.0840)	var16	3.4174 (0.0861)	var16	3.3761(0.0818)
var4	−0.8788 (0.0254)	var4	−0.8825 (0.0234)	var32	1.0786(0.0290)
var32	1.0590 (0.0309)	var32	1.0733 (0.0329)	var4	−0.8960(0.0247)
var17	−2.1169 (0.0672)	var17	−2.1592(0.0709)	var17	−2.1685(0.0618)
var6	0.3825 (0.0139)	var6	0.3862 (0.0134)	var12	−0.7668(0.0281)
var12	−0.7503 (0.0278)	var12	−0.7624 (0.0281)	var6	0.3887(0.0145)
var34	−0.9239 (0.0355)	var34	−0.9335 (0.0344)	var34	−0.9568(0.0367)
var25	0.4004 (0.0206)	var25	0.4082 (0.0206)	var25	0.4065(0.0214)

6 Discussion

This paper has proposed a simple, practical and efficient MCMC algorithm for Bayesian analysis of big data. The proposed algorithm has two innovations. First, it provides a simple and practical way to aggregate subposteriors to approximate the full data posterior. Second, it suggests to implement the Pop-SAMC algorithm to simulate from each subposterior. Since the whole algorithm consists of two levels of parallel, data parallel and simulation parallel, it is called the Double Parallel Monte Carlo algorithm. Theoretically, we have shown that the Double Parallel algorithm can produce a good approximation to the full data posterior distribution. Empirically, we have demonstrated that the results produced by the Double Parallel algorithm agree well with those generated from the full data posterior, while enabling massive speed-ups in computational time.

Compared to Consensus Monte Carlo, the Double Parallel algorithm has two advantages. First, it is more efficient in terms of quantities of effective samples generated in single unit CPU time. Recall that the Double Parallel algorithm converts the subset posterior samples to full-data posterior samples by simply recentering, while a weighted average applies for Consensus Monte Carlo. Second, it relaxes the requirement for the asymptotic normality of the posterior distributions by assuming the Laplace regularity for the log-likelihood function. This relaxation may explain why the Double Parallel algorithm outperforms Consensus Monte Carlo, as shown in Tables 1, 2 and 4, for our examples.

The Double Parallel algorithm works based on Laplace’s method, but it can also cover some problems that are traditionally treated as discrete, such as variable selection problems. As shown in Section 5, these problems can be treated as continuous by imposing a local shrinkage prior on the space of variable coefficients. A further extension of the proposed algorithm to general discrete parameter space will be of great interest.

Appendix A

Proof of Theorem 1

Under conditions A1 and A2, we can expand the mean of each subposterior at the corresponding MLE, θ̂^(j), as follows:

$${\mathit{\mu }}^{(j)}=E_{{\stackrel{\sim }{\pi }}_j}(\mathit{\theta })={\hat{\mathit{\theta }}}^{(j)}+\frac{{\hat{I}}^{(j)-1}}{n}[{\frac{\partial \text{ log }g(\mathit{\theta })}{\partial \mathit{\theta }}|}_{{\hat{\mathit{\theta }}}^{(j)}}-\frac{1}{2}{\hat{H}}^{(j)}{\hat{I}}^{(j)-1}]+O(n^{-2}),$$

where π̃_j = π̃(θ|X_[j]), ${\hat{I}}^{(j)}=-\frac{1}{m}{\frac{{\partial }^2\text{ log }f({\mathit{X}}_{[j]}|\mathit{\theta })}{\partial \mathit{\theta }\partial {\mathit{\theta }}^T}|}_{\mathit{\theta }={\hat{\mathit{\theta }}}^{(j)}},{\hat{H}}^{(j)}=-\frac{1}{m}{\frac{{\partial }^3\mathit{\text{ log }}f({\mathit{X}}_{[j]}|\mathit{\theta })}{\partial \mathit{\theta }\partial {\mathit{\theta }}^T\partial \mathit{\theta }}|}_{\mathit{\theta }={\hat{\mathit{\theta }}}^{(j)}}$, and Ĥ^(j) Î^(j)−1 is a vector whose rth element equals ${\sum }_{\mathit{\text{st}}}{\hat{H}}_{\mathit{\text{rst}}}^{(j)}{\hat{I}}_{\mathit{\text{st}}}^{(j)-1}$. To simplify the notation, we denote ${\hat{I}}^{(j)-1}[{\partial \text{ log }g(\mathit{\theta })/\partial \mathit{\theta }|}_{{\hat{\mathit{\theta }}}^{(j)}}-\frac{1}{2}{\hat{H}}^{(j)}{\hat{I}}^{(j)-1}]$ by ν^(j). Here we would like to mention that although A1 and A2 are directly associated with the Laplace approximation of the full posterior, i they can also lead to the conditions that guarantee the Laplace approximation of each subposterior, as long as m goes to ∞ with n.

Moreover, for each θ̂^(j), we have

$${\hat{\mathit{\theta }}}^{(j)}={\mathit{\theta }}^{\ast }+\frac{{\mathit{\xi }}^{(j)}}{\sqrt{m}}+O_p(m^{-1}),$$

where

$${\mathit{\xi }}^{(j)}=\frac{1}{\sqrt{m}}{I}^{-1}\sum _{i=1}^m\frac{\partial \text{ log }f(X_{\mathit{\text{ji}}}|{\mathit{\theta }}^{\ast })}{\partial \mathit{\theta }},I=-E_{\mathit{X}|{\mathit{\theta }}^{\ast }}\frac{{\partial }^2\text{ log }f(\mathit{X}|{\mathit{\theta }}^{\ast })}{\partial \mathit{\theta }\partial {\mathit{\theta }}^T}.$$

Therefore, the mean of the mixture distribution π̃(θ|X) is given by

$$E_{\stackrel{\sim }{\pi }}(\mathit{\theta })=\frac{1}{k}\sum _{j=1}^k{\mathit{\mu }}^{(j)}={\mathit{\theta }}^{\ast }+\frac{1}{k}\sum _{j=1}^k\frac{{\mathit{\nu }}^{(j)}}{n}+\frac{1}{k}\sum _{j=1}^k\frac{{\mathit{\xi }}^{(j)}}{\sqrt{m}}+O_p(m^{-1})+O(n^{-2}).$$

We can also implement a similar procedure on the full data posterior π(θ|X) and obtain

$$E_{\pi }(\mathit{\theta })={\mathit{\theta }}^{\ast }+\frac{\mathit{\nu }}{n}+\frac{\mathit{\xi }}{\sqrt{n}}+O_p(n^{-1})+O(n^{-2}),$$

where $\mathit{\nu }={\hat{I}}^{-1}[{\partial \text{ log }g(\mathit{\theta })/\partial \mathit{\theta }|}_{\hat{\mathit{\theta }}}-\frac{1}{2}\hat{H}{\hat{I}}^{-1}],\mathit{\xi }=\frac{1}{\sqrt{n}}{I}^{-1}{\sum }_{i=1}^n\frac{\partial \text{ log }f(X_i|{\mathit{\theta }}^{\ast })}{\partial \mathit{\theta }},\hat{I}=-\frac{1}{n}{\frac{{\partial }^2\text{ log }f(\mathit{X}|\mathit{\theta })}{\partial \mathit{\theta }\partial {\mathit{\theta }}^T}|}_{\mathit{\theta }=\hat{\mathit{\theta }}},\hat{H}=-\frac{1}{m}{\frac{{\partial }^3\mathit{\text{ log }}f(\mathit{X}|\mathit{\theta })}{\partial \mathit{\theta }\partial {\mathit{\theta }}^T\partial \mathit{\theta }}|}_{\mathit{\theta }=\hat{\mathit{\theta }}}$, and θ̂ denotes the MLE of θ calculated with the full dataset. By noting that $\frac{\mathit{\xi }}{\sqrt{n}}=\frac{1}{k}{\sum }_{j=1}^k\frac{{\mathit{\xi }}^{(j)}}{\sqrt{m}}$ and ν are O(1), m < n, equation (4) in Theorem 1 can thereby be verified:

$$E_{\stackrel{\sim }{\pi }}(\mathit{\theta })-E_{\pi }(\mathit{\theta })=\frac{1}{k}\sum _{j=1}^k\frac{{\mathit{\nu }}^{(j)}}{n}+O_p(m^{-1})-\frac{\mathit{\nu }}{n}+O_p(n^{-1})+O(n^{-2})=O(n^{-1})+O_p(m^{-1})+O(n^{-1})+O_p(n^{-1})+O(n^{-2})=O_p(m^{-1}).$$

The variance of each subposterior can be approximated as follows:

$${\mathit{\text{Var}}}_{{\stackrel{\sim }{\pi }}_j}(\mathit{\theta })=\frac{{\hat{I}}^{(j)-1}}{n}+O(n^{-2}).$$

Therefore, the variance of the mixture distribution π̃(θ|X) is given by

$${\mathit{\text{Var}}}_{\stackrel{\sim }{\pi }}(\mathit{\theta })=\frac{1}{k}\sum _{j=1}^k{\mathit{\text{Var}}}_{{\stackrel{\sim }{\pi }}_j}(\mathit{\theta })=\frac{1}{k}\sum _{j=1}^k\frac{{\hat{I}}^{(j)-1}}{n}+O(n^{-2}).$$

In addition, we have

$${\mathit{\text{Var}}}_{\pi }(\mathit{\theta })=\frac{{\hat{I}}^{-1}}{n}+O(n^{-2}).$$

Since Î^(j) = I + o_p(1) and Î = I + o_p(1), we further have Î^(j)−1 = I⁻¹ + o_p(1) and Î⁻¹ = I⁻¹ + o_p(1). The equation (5) in Theorem 1 can thereby be verified:

$${\mathit{\text{Var}}}_{\stackrel{\sim }{\pi }}(\mathit{\theta })-{\mathit{\text{Var}}}_{\pi }(\mathit{\theta })=\frac{1}{k}\sum _{j=1}^k\frac{{\hat{I}}^{(j)-1}}{n}-\frac{{\hat{I}}^{-1}}{n}+O(n^{-2})=\frac{I^{-1}}{n}+o_p(n^{-1})-\frac{I^{-1}}{n}+o_p(n^{-1})+O(n^{-2})=o_p(n^{-1}).$$

Finally, for the square of the Wasserstein distance of order 2, we have

$$|d^2(\stackrel{\sim }{\pi }(\mathit{\theta }|\mathit{X}),{\delta }_{{\mathit{\theta }}^{\ast }})-d^2(\pi (\mathit{\theta }|\mathit{X}),{\delta }_{{\mathit{\theta }}^{\ast }})|=|{\int }_{\mathrm{\Theta }}{\Vert \mathit{\theta }-{\mathit{\theta }}^{\ast }\Vert }_2^2\stackrel{\sim }{\pi }(\mathit{\theta }|\mathit{X})d\mathit{\theta }-{\int }_{\mathrm{\Theta }}{\Vert \mathit{\theta }-{\mathit{\theta }}^{\ast }\Vert }_2^2\pi (\mathit{\theta }|\mathit{X})d\mathit{\theta }|=|{\Vert E_{\stackrel{\sim }{\pi }}(\mathit{\theta })-{\mathit{\theta }}^{\ast }\Vert }_2^2+\mathit{\text{tr }}({\mathit{\text{Var}}}_{\stackrel{\sim }{\pi }}(\mathit{\theta }))-{\Vert E_{\pi }(\mathit{\theta })-{\mathit{\theta }}^{\ast }\Vert }_2^2-\mathit{\text{tr }}({\mathit{\text{Var}}}_{\pi }(\mathit{\theta }))|\le |{\Vert E_{\stackrel{\sim }{\pi }}(\mathit{\theta })-E_{\pi }(\mathit{\theta })+E_{\pi }(\mathit{\theta })-{\mathit{\theta }}^{\ast }\Vert }_2^2-{\Vert E_{\pi }(\mathit{\theta })-{\mathit{\theta }}^{\ast }\Vert }_2^2|+|\mathit{\text{tr }}({\mathit{\text{Var}}}_{\stackrel{\sim }{\pi }}(\mathit{\theta })-{\mathit{\text{Var}}}_{\pi }(\mathit{\theta }))|\le {\Vert E_{\stackrel{\sim }{\pi }}(\mathit{\theta })-E_{\pi }(\mathit{\theta })\Vert }_2^2+2{\Vert E_{\stackrel{\sim }{\pi }}(\mathit{\theta })-E_{\pi }(\mathit{\theta })\Vert }_2{\Vert E_{\pi }(\mathit{\theta })-{\mathit{\theta }}^{\ast }\Vert }_2+o_p(n^{-1})=O_p(m^{-2})+2O_p(m^{-1})O_p(n^{-1/2})+o_p(n^{-1})=o_p(n^{-1}),$$

where the last equality follows from the fact n = o(m²). This verifies the equation (6) in Theorem 1.