Variable selection for partially linear models via Bayesian subset modeling with diffusing prior

Abstract

Most existing methods of variable selection in partially linear models (PLM) with ultrahigh dimensional covariates are based on partial residuals, which involve a two-step estimation procedure. While the estimation error produced in the first step may have an impact on the second step, multicollinearity among predictors adds additional challenges in the model selection procedure. In this paper, we propose a new Bayesian variable selection approach for PLM. This new proposal addresses those two issues simultaneously as (1) it is a one-step method which selects variables in PLM, even when the dimension of covariates increases at an exponential rate with the sample size, and (2) the method retains model selection consistency, and outperforms existing ones in the setting of highly correlated predictors. Distinguished from existing ones, our proposed procedure employs the difference-based method to reduce the impact from the estimation of the nonparametric component, and incorporates Bayesian subset modeling with diffusing prior (BSM-DP) to shrink the corresponding estimator in the linear component. The estimation is implemented by Gibbs sampling, and we prove that the posterior probability of the true model being selected converges to one asymptotically. Simulation studies support the theory and the efficiency of our methods as compared to other existing ones, followed by an application in a study of supermarket data.

1. Introduction

Semiparametric model attracts considerable attention in the literature since it retains the interpretability of the parametric models and keeps some flexibility of the nonparametric models. In this paper, we study a type of commonly used semiparametric model, the partially linear model (PLM). The PLM assumes that, the response Y depends both linearly on some covariates $X\in {ℝ}^p$ of interest, and nonparametrically on another univariate continuous covariate U defined on [0, 1]. Suppose that the observed data {(Y_i, X_i, U_i)}, i ∈ {1, …, n}, is a random sample from the following PLM

$$Y=f(U)+X^{\top }\beta +ϵ,ϵ~\mathcal{N}\left(0,{\sigma }^2\right).$$ (1)

This PLM specifies a parsimonious linear function in the parametric part, while allowing a nonparametric component to be unconstrained and subject to empirical estimation. In this paper, a new one-step Bayesian approach is proposed to select variables for PLM with ultrahigh dimensional covariate X, that is ln p = o(n). Specifically, our proposed method simplifies the procedure by avoiding estimating the infinite dimensionality brought by the nonparametric component and results in sparsity in the linear component.

The estimation procedure for PLM with fixed dimension p of X has been extensively studied. Engle et al. used the penalized least squares method to estimate β and the nuisance function f(·) simultaneously by adding a penalty on the roughness of f(·), which was referred as the partial smoothing splines [4, 7, 12, 23]. Since β is of primary interest, some other methods brilliantly avoid the estimation of f(·). For example, Robinson [24] introduced a profile least squares estimator based on the idea of partial residual, which later became one of the commonly used approaches to eliminate the nonparametric component in PLM. Another type of approach to eliminate the nonparametric component is the difference-based method [27, 29]. It estimates the coefficients in linear component by taking differences of the ordered observations. The resulting estimator is proven to be asymptotically efficient under finite dimension. See Section 2.1 for more details about the difference-based method.

Variable selection for PLM can be accomplished by adding another penalty function on β to the loss function of the aforementioned partial smoothing splines method. The least absolute shrinkage and selection operator (LASSO) [25], the nonnegative garrote [2, 31], the smoothly clipped absolute deviation (SCAD) [8], the elastic net [34], and the minimum concave penalty (MCP) [32] are all among popular choices of penalty functions. Xie and Huang [28] used the SCAD penalty to achieve the sparsity in the linear part and used the polynomial splines to estimate the nonparametric component simultaneously. The resulting estimator $\widehat{\beta }$ was shown to be consistent with $p=o(\sqrt{n})$. An alternative is to use a two-step procedure, in which Y and X are first regressed on U separately to get partial residuals, then the variable selection is further applied on the transformed model. For example, the consistency for both the linear and the nonparametric components has been well studied by Zhu et al. [33] under the regime of $p=o(\sqrt{n})$. Liang and Li [19] discussed this approach in the presence of measurement errors. More recently, Liu et al. [20] proposed selection procedure via recursively test on the partial correlation among the partial residuals and among the covariates when ln p = o(n). The method was referred as the thresholded partial correlation on partial residuals (TP-CPR). However, to the best of our knowledge, there is nearly no literature on variable selection in the high-dimensional setting based on the extension of difference-based method.

Bayesian approach puts priors on the parameters and the model space, and selects the model with the highest posterior probability. There has been multiple development for variable selection using Bayesian approach with linear and generalized linear models. George and McCulloch [11] proposed a milestone method of Bayesian variable selection via stochastic search. They introduced a latent binary vector to indicate the inclusion of variables in linear models, and then placed a mixture spike and slab prior on each coefficient conditioning on this latent vector. Following this approach, many other selection procedures with similar structure have been proposed. The distinction between them is mostly in the form of the spike and slab priors, or in the form of the prior on the model space. To alleviate the difficulty in choosing specific prior parameters, several approaches have been proposed, see [10, 14, 30]. However, these papers focused on small-scaled questions and did not discuss any possible extension to the high-dimensional setting. More recently, Ishwaran and Rao [15] established the oracle property of the posterior mean as n converges to infinity with fixed p under certain conditions on the prior variances for linear models. Johnson and Rossell [16] proved selection consistency under p = O(n) for a non-local prior in linear model settings. Liang et al. [18] proposed a point-mass spike prior with a slab prior depending on the model size, and proved the posterior consistency under ln p = o(n) in generalized linear models, but the corresponding conditions are relatively strong. Additionally, the step-wise estimation procedure is not efficient. Narisetty and He [21] also used Gaussian prior but argued the prior should be sample-size dependent, referred to as Bayesian shrinking and diffusing priors (BASAD), and obtained strong selection consistency when ln p = o(n) for linear models under mild assumptions. However, BASAD is not computationally practical for large-p problems, since it requires to update β from a p-dimensional multivariate normal distribution in each iteration. Recently Narisetty et al. [22] proposed Skinny Gibbs (SG) algorithm to address this computation issue via sparsifying the precision matrix. They referred to this kind of update as Skinny Gibbs (SG) and argued that it is a scalable method, namely the required computation time grows approximately linearly in p. The selection consistency was proved for the logistic regression. While spike and slab priors have been widely used in applications for its attractive interpretability, the theory for spike and slab models has not caught up with the applications. Again, all the aforementioned papers focused on linear or generalized linear models, and the corresponding work on semiparametric or nonparametric models under high dimensional setting is limited.

In this paper, we propose a Bayesian subset selection procedure for the partially linear model. We incorporate the difference-based method in the prior for the nonparametric component. For the parametric component, we adopt a modified version of Bayesian shrinking and diffusing priors (BASAD) [21] and propose the novel Bayesian subset modeling with diffusing prior (BSM-DP). We use a normal distribution with a diverging variance as the slab prior and a normal distribution with a small variance as the spike prior. Differently from BASAD, the response variable in our model only depends on the active covariates. This conveniently allows us to sample coefficients separately for the active and the inactive sets during the estimation. In fact, the spike prior has no impact on the theoretical result, so any proposal including a point mass will work. As a practical note, we recommend a Gaussian distribution with a small variance, which allows more flexibility for the Markov chain to explore the model space, and hence avoiding local trap. As a result, the proposed methods are more computationally efficient than BASAD. We also notice that the Skinny Gibbs (SG) [22] is a special case of BSM-DP when the variance of spike prior is set to be proportional to 1/n. Their original paper [22] discussed logistic regression only. We establish the selection consistency for the parametric component in partially linear models when ln p = o(n) under mild conditions.

The rest of the paper is organized as follows. In Section 2 we present the Bayesian subset modeling with diffusing prior (BSM-DP) and discuss variable selection for partially linear model, followed by the estimation procedure, regularity conditions and theoretical results. Performances of several numerical studies are presented in Section 3 to demonstrate reliability of the proposed model. We further apply the proposed method on the supermarket data set. Proofs for lemmas and theorems are given in Section 4, followed by discussions in Section 5.

2. Bayesian subset modeling with diffusing prior

2.1. Model and notation

Suppose that {(Y_i, X_i, Ui)}, i ∈ {1, …, n} is a random sample from PLM (1) with high-dimensional covariates $X\in {ℝ}^{p_n}$ and univariate covariate U ∈ [0, 1], where we use p_n to emphasize that the number of variables is allowed to diverge with sample size n. Assume that the random error ϵ is independent of (X^⊤, U) and each observation X_i has the same distribution with mean 0 and covariance Σ. Denote f(U_i) as α_i, and α = (α₁, …., α_n)^⊤ as a vector with size n. Notation Y is the corresponding size n vector, and X is the design matrix with size n × p_n.

We will propose a prior for the nonparametric function (i.e. the α) in our proposed Bayesian subset selection based on difference-based method. Assume the observation {(Y_i, X_i, Ui)}_1≤i≤n is ordered by the increasing order of {Ui}_1≤i≤n. The difference in observed value for contingent Y can be written as

$$Y_i-Y_{i-1}=\left\{f\left(U_i\right)-f\left(U_{i-1}\right)\right\}+{\left(X_i-X_{i-1}\right)}^{\top }\beta +{ϵ}_i-{ϵ}_{i-1},i\in \left\{2,\dots ,n\right\}.$$

If U_i−1 and U_i are close and f(·) is smooth enough, f(U_i) should also be close to f(U_i−1). So the nonparametric part tends to be cancelled out. In this case, the ordinary least squares estimate can be applied on the differenced data, as long as X is not perfectly correlated with U. Define the mth higher order difference sequence to be {d_i}_{i∈{1,…,m+1}}, which satisfies ${\sum }_{i=1}^{m+1}{d}_i=0$ and ${\sum }_{i=1}^{m+1}{d}_i^2=1$. So the mth order difference operation reduces the sample size to n – m by defining

$$Y_i^{*}=C^{1/2}\sum _{t=1}^{m+1}{d}_t{Y}_{i+m+1-t},X_i^{*}=C^{1/2}\sum _{t=1}^{m+1}{d}_t{X}_{i+m+1-t},{\delta }_i=C^{1/2}\sum _{t=1}^{m+1}{d}_{t}f\left(U_{i+m+1-t}\right),{\omega }_i=C^{1/2}\sum _{t=1}^{m+1}{d}_t{ϵ}_{i+m+1-t},$$

for i ∈ {1, …, n − m}, where C is some positive constant. Define the difference matrix D as

$$D=\left(\begin{array}{ccccccc}{d}_{m+1}& \dots & d_1& 0& \dots & \dots & 0\\ 0& d_{m+1}& \dots & d_1& 0& \dots & 0\\ ⋮& \ddots & \ddots & \ddots & \ddots & ⋮& ⋮\\ 0& \dots & 0& d_{m+1}& \dots & d_1& 0\\ 0& \dots & \dots & 0& d_{m+1}& \dots & d_1\end{array}\right)\in {ℝ}^{(n-m)\times n}.$$ (2)

Therefore the PLM (1) can be rewritten in matrix form as

$$Y^{*}=X^{*}\beta +\delta +\omega ,$$

where $Y^{*}=C^{1/2}DY\in {ℝ}^{n-m}$, $X^{*}=C^{1/2}DX\in {ℝ}^{(n-m)\times p_n}$, $\delta =C^{1/2}D\mathit{\alpha }\in {ℝ}^{n-m}$, $\omega =C^{1/2}D\mathit{ϵ}\in {ℝ}^{n-m}$. Under some smoothness conditions on f(·) with fixed p, Yatchew [29] and Wang et al. [27] showed that the ordinary least square estimator $\widehat{\beta }={\left(X^{*\top }{X}^{*}\right)}^{-1}{X}^{*\top }{Y}^{*}$ is asymptotically efficient when m → ∞, if X and U are independent. This indicates that the effect of the nonparametric component is negligible after applying a high order difference operation on the data.

In the literature, U_i’s are either from a fixed design e.g., U_i = i/n, or observations from a distribution on [0, 1] with density function bounded away from 0. In this paper, we only consider the case when X and U are independent under a dense design with a bounded max_2≤i≤n |U_i − U_i−1|.

We use X_j as the notation for the jth covariate. A size-p_n latent binary random vector is introduced as γ. The jth entry γ_j indicates whether X_j is included in the model (1 = present, 0 = not present). Therefore, the model space is fully specified by γ, and we use γ and $\mathcal{M}$ as notations for models interchangeably. The true model is denoted as $\mathcal{A}$. The cardinality of model $\mathcal{M}$, denoted by $\left|\mathcal{M}\right|$, is the size of the model. Consequently, if ${\beta }_{\mathcal{M}}$ is the subvector of β with size $\left|\mathcal{M}\right|$, $X_{\mathcal{M}}$ is the submatrix of X with respect to model $\mathcal{M}$, and ${\Sigma }_{\mathcal{M}}$ is the $\left|\mathcal{M}\right|\times \left|\mathcal{M}\right|$ covariance matrix for $X_{\mathcal{M}}$. Other notations used in the paper are unified as follows.

• Model operation: ${\mathcal{M}}_1\wedge {\mathcal{M}}_2$ and ${\mathcal{M}}_1\vee {\mathcal{M}}_2$ are defined as the intersection and union of model ${\mathcal{M}}_1$ and ${\mathcal{M}}_2$, for example ${\mathcal{M}}_1\wedge {\mathcal{M}}_2=\{i:i\in {\mathcal{M}}_1\text{\hspace{0.17em}and\hspace{0.17em}}i\in {\mathcal{M}}_2\}$.

• Rate: a_n ⪯ b_n or b_n ⪰ a_n means a_n = O(b_n), a_n ≺ b_n or b_n ≻ a_n means a_n = o(b_n). And a_n ~ b_n refers a_n/b_n → c for some positive constant.

• Matrix and matrix operation: n × n identity matrix is denoted as I_n. For a matrix M, ∥M∥ is the spectral norm, which is the largest singular value of M. The Moore-Penrose inverse of M is denoted by M⁺, which is the unique generalized inverse. And if M is a positive definite matrix we use λ_min(M) and λ_max(M) as the notation for the minimum and the maximum eigenvalue of M.

Here and hereafter, the densities are conditional on X and U. The working model for variable selection in the partially linear model in (1) via Bayesian subset modeling with diffusing prior (BSM-DP) is proposed as

$$\begin{gathered} \pi (Y\mid \alpha ,\gamma ,\beta ,\sigma )=\mathcal{N}\left(\alpha +X_{\gamma }{\beta }_{\gamma },{\sigma }^2{I}_n\right), \\ \pi \left({\beta }_j\mid \gamma ,\sigma \right)=\{\begin{array}{ll}\mathcal{N}\left(0,{\sigma }_{1n}^2{\sigma }^2\right)\hfill & {\gamma }_j=1\hfill \\ \mathcal{N}\left(0,{\sigma }_0^2{\sigma }^2\right)\hfill & {\gamma }_j=0\hfill \end{array},j\in \left\{1,\dots ,p_n\right\}, \\ \pi (\gamma )=\prod _{j=1}^{p_n}{q}_n^{{\gamma }_j}{\left(1-q_n\right)}^{1-{\gamma }_j},\pi (\alpha \mid \sigma )=\mathcal{N}\left(0,{\sigma }^2{\Sigma }_{0n}\right),\pi \left({\sigma }^2\right)=IG\left(a_0,b_0\right), \end{gathered}$$ (3)

where, ${\Sigma }_{0n}={\left\{{\left(I_n-C{D}^{\top }D\right)}^{-1}-I_n\right\}}^{+}$, D is the difference matrix defined in (2).

We choose the classical Inverse Gamma distribution as the prior for σ² as it is the most commonly used conjugate prior. Other choices of prior could be used, and it can be shown that Theorem 5 applies to a wider family of priors, including some commonly choices like improper non-informative prior and the class of folded-noncentral-t prior (see Remark 8). Meanwhile, independent Bernoulli distribution with probability q_n is used as the prior for each γ_j. So the preliminary marginal inclusion probability for each variable is q_n. It is natural to assume that when the dimension p_n diverges with the sample size, q_n should converge at some rate to 0. Each β_i has a mixture normal distribution. Conditioning on γ_j = 1, β_j has a normal distribution with a relative large variance ${\sigma }_{1n}^2{\sigma }^2$. This corresponds to a very wide and flat distribution, usually referred to as a slab prior. We call it the diffusing prior as named in [21]. Within its variance, σ_1n depends on the sample size, and diverges at some certain rate when the sample size goes to infinity. Conditioning on γ_j = 0, β_j has a normal distribution with variance ${\sigma }_0^2{\sigma }^2$. As the choice of ${\sigma }_0^2$ would not influence the asymptotic results, it can be chosen depending on sample size or simply as a fixed value.

With a partially linear model, we will need to accommodate the nonparametric part. A conjugate prior of normal distribution with a semi-definite covariance matrix σ²Σ_0n is proposed for α = (f(U₁), …, f(U_n)). The covariance matrix σ²Σ_0n is further taken as a function of the difference matrix D thus to eliminate the effect of nonparametric function. More intuitions about the choice of Σ_0n will be discussed later. The error term is assumed to be normally distributed. Therefore, conditional on the latent indicator γ, coefficients β, nonparametric component α, and the variance for the error σ, Y has a normal distribution.

Remark 1. (Comparison with BASAD [21] and SG [22]). As mentioned earlier, the inclusion of γ in the conditional distribution of Y distinguishes our model from BASAD. This difference allows us to sample separately for the active and the inactive groups.

In our working model (3), the response variable Y is conditioned on γ, hence only depends on the active covariates X_γ. But in BASAD, Y depends on both the active and nonactive part of the covariates. As a result, the full conditional distribution for β_γ and ${\beta }_{{\gamma }^c}$ are not independent in BASAD. Therefore, to update β in MCMC, each iteration requires sampling a size-p vector from a multivariate normal distribution. This will increase the computational time quickly under large p. On the other hand, the full conditional distribution for β_γ and ${\beta }_{{\gamma }^c}$ are independent in our proposal. So in each iteration, we only need to sample a size-|γ| vector from a multivariate normal distribution and sample (p − |γ|) scalars from independent univariate normal distributions. The current active model size |γ| is usually small after several iterations if the true model is sparse. In fact, just as SG, the proposed BSM-DP is scalable in high dimensional problems, which means that the computation time growing approximately linearly with the dimension p. The computational complexity for each iteration in the estimation procedure is n(p ∨ |γ|² ∨ n²). We have also validated this claim in the simulation study for PLM and more simulation studies about linear models as compared with BASAD and SG in the supplementary material. It can be shown that SG is equivalent to our Bayesian subset modelling by taking the variance of spike prior to be ${\sigma }_0^2={\left(n+{\tau }_{0n}^{-2}\right)}^{-1}$, where ${\tau }_{0n}^2$ is the variance for spike prior in SG.

2.2. Estimation procedure

Gibbs sampling is used to update our parameters iteratively. In each iteration, we draw samples from those full conditional distributions.

1. Update γ_k from a Bernoulli distribution:

   Full conditional distribution of γ_k is a Bernoulli distribution with probability Pr(γ_k = 1|β, α, σ², Y, γ_−k) = p₁/ (p₁ + p₂), and

   $$\frac{p_1}{p_2}=\frac{q_n{\sigma }_0}{\left(1-q_n\right){\sigma }_{1n}}\text{exp}\left\{-\frac{{\beta }_k^2}{2{\sigma }_{1n}^2{\sigma }^2}+\frac{{\beta }_k^2}{2{\sigma }_0^2{\sigma }^2}+{\beta }_k{X}_k^{\top }\left(Y-\alpha -X_{{\widehat{\mathcal{A}}}_k}{\beta }_{{\widehat{\mathcal{A}}}_k}\right)/{\sigma }^2-{\beta }_k^2{X}_k^{\top }{X}_k/\left(2{\sigma }^2\right)\right\},$$

   where X_i is the ith column of X, and the index ${\widehat{\mathcal{A}}}_k$ is the collection of current active covariates after removing the kth covariate, that is ${\widehat{\mathcal{A}}}_k=\left\{i:{\gamma }_i=1,i\ne k\right\}$.

2. Update β from multivariate normal distributions:

   In each iteration, we divide β into the active group and the inactive group based on the current γ. Denote $\widehat{\mathcal{A}}$ as the collection of covariates with γ_j = 1, and $\widehat{\mathcal{I}}$ as the collection of covariates with γ_j = 0. Rewrite $\beta =\left({\beta }_{\widehat{\mathcal{A}}},{\beta }_{\widehat{\mathcal{I}}}\right)$, and we can update those two groups separately.

   1. Update the active group $\widehat{\mathcal{A}}:{\beta }_{\widehat{\mathcal{A}}}~\mathcal{N}\left(V{X}_{\widehat{\mathcal{A}}}^{\top }(Y-\alpha ),{\sigma }^{2}V\right)$, where $V={\left(X_{\widehat{\mathcal{A}}}^{\top }{X}_{\widehat{\mathcal{A}}}+\frac{1}{{\sigma }_{1n}^2}{I}_{\left|\widehat{\mathcal{A}}\right|}\right)}^{-1}$.
   2. Update the inactive group $\widehat{\mathcal{I}}:{\beta }_{\widehat{I}}~\mathcal{N}\left(0,{\sigma }_0^2{\sigma }^2{I}_{\left|\widehat{\mathcal{I}}\right|}\right)$.

3. Update σ² from an Inverse Gamma distribution:

   σ² ~ IG(a, b), where

   $$a=a_0+\left(n+p_n\right)/2,$$

   $$b=b_0+{\beta }_{\widehat{\mathcal{A}}}^{\top }{\beta }_{\widehat{\mathcal{A}}}/\left(2{\sigma }_{1n}^2\right)+{\beta }_{\widehat{\mathcal{I}}}^{\top }{\beta }_{\widehat{\mathcal{I}}}/\left(2{\sigma }_0^2\right)+{\left(Y-\alpha -X_{\widehat{\mathcal{A}}}{\beta }_{\widehat{\mathcal{A}}}\right)}^{\top }\left(Y-\alpha -X_{\widehat{\mathcal{A}}}{\beta }_{\widehat{\mathcal{A}}}\right)/2+{\alpha }^{\top }{\Sigma }_{0n}\alpha /2.$$
4. Update α from a multivariate normal distribution:

   $$\alpha ~\mathcal{N}\left({\Sigma }_{\alpha n}\left(Y-X_{\widehat{\mathcal{A}}}{\beta }_{\widehat{\mathcal{A}}}\right),{\sigma }^2{\Sigma }_{\alpha n}\right),\text{where\hspace{0.17em}}{\Sigma }_{\alpha n}={\left({\Sigma }_{0n}^{+}+I_n\right)}^{-1},\$\text{furthermore\hspace{0.17em}by\hspace{0.17em}Condition\hspace{0.17em}}A,{\Sigma }_{\alpha n}=I_n-C{D}^{\top }D.$$

In the literature, the nonparametric function f(·) is usually assumed to be smooth, which means f(x) and f(y) should be close if x and y are close enough. This dependency among f(U₁), f(U₂), …, f(U_n) suggests that the covariance matrix Σ_0n of α has to be a dense matrix. Here we take Σ_0n to be Σ_0n = {(I_n – CD^⊤D)⁻¹ − I_n}⁺ where C is some positive constant and D is the difference matrix defined in (2). We will show the reason for this specific choice of Σ_0n in Remark 2. Fig. 1. shows the intuitive structure of difference matrix D, the matrix Σ_αn used for the update of α, and covariance matrix for the prior Σ_0n when constant C = 0.6 with sample size n = 200 and difference order m = 20. As demonstrated in the figure, $D\in {ℝ}^{(n-m)\times n}$ is an general upper triangular band matrix with bandwidth m. The update matrix Σ_αn is also a band matrix with bandwidth m. The covariance matrix for the prior of the nonparametric component ${\Sigma }_{0n}\in {ℝ}^{n\times n}$ is a dense matrix with positive larger values near the diagonal, then decays gradually to 0 and negative when moving further. The reason why it has negative off-diagonal values is because the difference sequence {d_i}_1≤i≤m+1 is standardized to be centered at 0. Theoretically, we do require the difference order m, which is also the bandwidth, goes to infinity as n → ∞ at some slow rate. In this way, the effect of the nonparametric component can be removed without over-smoothing the nonparametric function f(·), so the selection consistency for the linear component holds.

Fig. 1.

Visualization to display the magnitude of values in the difference matrix D, the covariance matrix Σ_αn used for the update of α and the covariance matrix for prior Σ_0n. All plots are taking constant C = 0.6, sample size n = 200 and difference order m = 20. On the graph, red color indicates positive values on the the corresponding locations of the matrix, and purple color represents negative values.

2.3. Selection procedure

In the typical Bayesian variable selection approach, the model with the highest posterior is selected as the final model, referred to as maximum a posterior model (MAP): $\widehat{\mathcal{M}}={\text{argmax}}_{\mathcal{M}}\text{Pr}(\gamma =\mathcal{M}\mid Y)$. With the spike and slab prior, the posterior of the model space is usually reflected by the posterior probability of the latent variable γ. Alternatively, another way is to consider the marginal probability of Pr(γ_j = 1|Y). Specifically, one will select the jth covariate if Pr(γ_j = 1|Y) is equal to or greater than a certain threshold. A threshold of 0.5 is a natural choice. This is known as the median probability model (MPM). It has been shown that MPM has good predictive power [1]. Although it is likely that these two approaches may produce different results in practice, it can be shown those two selection methods are asymptotically the same under strong selection consistency, which will be shown in Section 2.4. Moreover, some other data-driven criteria could also be used in determining the threshold, e.g. AIC, BIC and EBIC [5].

2.4. Theoretical results

Variable selection procedures typically aim to achieve selection consistency, and under Bayesian framework, it means conditional on observed data, the probability of the true model $\mathcal{A}$ being selected goes to 1 in probability.

$$\text{Pr}(\widehat{\mathcal{M}}=\mathcal{A}\mid Y)\stackrel{P}{\to }1\text{ as\hspace{0.17em}}n\to \infty .$$

That is, the true model is selected consistently. Note that the posterior of model space is fully specified by γ. If the model is selected via MAP: $\widehat{\mathcal{M}}={\text{argmax}}_{\mathcal{M}}\text{Pr}(\gamma =\mathcal{M}\mid Y)$, then the selection consistency only requires that the posterior probability of the true model, i.e. $\text{Pr}(\gamma =\mathcal{A}\mid Y)$ is no less than that of any other models, i.e. $\text{Pr}(\gamma =\mathcal{M}\mid Y)$. But the difference in their posterior probabilities could still shrink to 0. In this paper, we will consider the following strong selection consistency

$$\text{Pr}(\gamma =\mathcal{A}\mid Y)\stackrel{P}{\to }1\text{ as\hspace{0.17em}}n\to \infty .$$

It indicates the difference for the posterior probabilities of the true model and any other model is 1. This nonzero difference indicates a stronger conclusion than selection consistency. We first present the following regularity conditions for the selection consistency of the linear component in the PLM, and we then start with the case when σ² is known as it provides intuitive interpretation for the proposed method.

Condition A (On the dimension and priors). The dimension p_n satisfies that ln(p_n) = o(n). The prior probability that a coefficient is nonzero q_n satisfies that q_n ~ 1/p_n. The variance for slab prior ${\sigma }_{1n}^2\to \infty$ as n → ∞, and $n{\sigma }_{1n}^2{\lambda }_1~p_n^{2+3\delta }$ for some δ > 0, where λ₁ is defined in Conditions C. The covariance for the prior of nonparametric component Σ_0n = {(I_n – CD^⊤D)⁻¹ − I_n}⁺, where C is a positive constant, with values no greater than min {1, 1/λ_max(D^⊤D)}, and D is the difference matrix defined in (2).

Condition B (Identifiability). There exists K > 1 + 4/δ such that

$${\Delta }_n(K)=\underset{\mathcal{M}:\left|\mathcal{M}\right|\le K\left|\mathcal{A}\right|,\mathcal{A}⊈\mathcal{M}}{\text{inf}}{\Vert \left(I-P_{\mathcal{M}}\right)X_{\mathcal{A}}^{*\top }{\beta }_{\mathcal{A}}\Vert }_2^2>{\sigma }^2\left|\mathcal{A}\right|(4+4\delta +\kappa )\text{ln\hspace{0.17em}}{p}_n,$$

where X* = C^1/2DX and the projection matrix $P_{\mathcal{M}}=X_{\mathcal{M}}^{*}{\left(X_{\mathcal{M}}^{*\top }{X}_{\mathcal{M}}^{*}\right)}^{-1}{X}_{\mathcal{M}}^{*\top }$.

Condition C (Regularity of the design). Define

$${\lambda }_1=\underset{\mathcal{M}:\left|\mathcal{M}\right|\le m_n+\left|\mathcal{A}\right|}{\text{min}}{\lambda }_{\text{min}}\left(\frac{1}{n}{X}_{\mathcal{M}}^{\top }{X}_{\mathcal{M}}\right),{\lambda }_2=\underset{\mathcal{M}:\mathcal{M}\subseteq \mathcal{A}}{\text{max}}{\lambda }_{\text{max}}\left(\frac{1}{n}{X}_{\mathcal{M}}^{\top }{X}_{\mathcal{M}}\right),$$

then ${\left({\lambda }_2/{\lambda }_1\right)}^{\left|\mathcal{A}\right|}⪯p_n^{\kappa }$, for some 0 < κ < δ, where m_n is defined in Condition D.

Condition D (On the true model). Let ${\beta }_{\mathcal{A}}$ consist of all nonzero elements of β. That is, $X_{\mathcal{A}}$ consists of all active predictors. The size of $\mathcal{A}$ satisfies that $\left|\mathcal{A}\right|=o\left(m_n\right),m_n=cn/\text{ln\hspace{0.17em}}{p}_n$, m_n = cn/ ln p_n, where c < δ/ {(4 + δ)(2 + δ)}. Further assume that $U_{\infty }={\text{max}}_{2\le i\le n}\left|U_i-U_{i-1}\right|=O\left(n^{-c_1}\right)$ for some 0 < c₁ ≤ 1.

Condition E (On the difference matrix). Let D be the difference matrix, as defined in (2). Denote $h_k={\sum }_{i=1}^{m+1-k}{d}_i{d}_{i+k}$, then the mth difference sequence d₁, …, d_m+1 satisfies,

$$\sum _{i=1}^{m+1}{d}_i=0,\sum _{i=1}^{m+1}{d}_i^2=1,\sum _{k=1}^m{h}_k^2=O\left(m^{-1}\right),1-d_1^2=O\left(m^{-1}\right).$$

Furthermore m → ∞, $m=o\left(n^{c_2}\right)$ for some 0 < c₂ < c₁ ≤ 1, where c₁ is defined in Condition D.

Condition F (On the nonparametric component f(·)). Suppose f(·) ∈ Λ^k(M) for some $k>\frac{1}{2\left(c_1-c_2\right)}$, where c₁ and c₂ is defined in Conditions D, E. The Lipschitz ball Λ^k(M) is defined as

$${\Lambda }^k(M)=\{f:\text{for\hspace{0.17em}all\hspace{0.17em}}0\le x,y\le 1,\left|f^{(i)}(x)\right|\le M,i\in \{0,\dots ,\lfloor k\rfloor -1\},\left|f^{\left(\lfloor k\rfloor \right)}(x)-f^{\left(\lfloor k\rfloor \right)}(y)\right|\le M|x-y{|}^{k^{\prime }}\},$$

where ⌊k⌋ is the largest integer less than k and k′ = k − ⌊k⌋.

The convergent and divergent rate of the parameters in the priors and the dimension of the variables are stated in Condition A. Identifiability Condition B is needed to distinguish active covariates out of spurious ones. C gives the regularity condition of the design matrix. Instead of requiring bounded eigenvalues, we will need the minimal eigenvalues to decay slower than some rate and the maximal eigenvalues to diverge slower than some rate. We would like to point that, if the size of the true model is not too large, the condition holds even with the extreme case when X is sampled from normal distribution with compound symmetric covariance matrix when correlation among predictors ρ → 1. Theoretically the model still works even under nearly perfectly correlated covariates. Condition D states the normality assumption for the error and we do allow infinitely many active variables.

Conditions E-F control the error in estimating the nonparametric component. Condition E is about the difference matrix. A sequence satisfying the above conditions, for example, is $d_1=\sqrt{\frac{m}{m+1}}$, $d_2=d_3=\cdots =d_{m+1}=-\sqrt{\frac{1}{m(m+1)}}$. As argued in [23] about partial smoothing spline method for PLM, higher ordered difference operation gives lower approximation error. We do assume m → ∞, so the approximation error becomes ignorable. In the estimation of the partial residuals, it requires that the nonparametric estimators of E(Y|U) and E(X|U) to converge sufficiently fast so that their substitutions in the OLS estimator do not affect its asymptotic distribution. Similarly, our upper bound for the growth rate of difference order m reflects this. Finally, the commonly used smoothness assumption for nonparametric nuisance function is stated in Condition F.

The first step is to derive posterior probability of any model $\mathcal{M}$.

Lemma 1. Under fixed σ², for any model $\mathcal{M}$, the posterior probability has the following explicit form:

$$\text{Pr}\left(\gamma =\mathcal{M}\mid Y,{\sigma }^2\right)\propto q_n^{\left|\mathcal{M}\right|}{T}_{\mathcal{M}}\text{exp}\left(-\frac{1}{2{\sigma }^2}{R}_{\mathcal{M}}\right),$$

where

$$R_{\mathcal{M}}=Y^{\top }\left\{{\Sigma }_{1n}-{\Sigma }_{1n}{X}_{\mathcal{M}}{\left(X_{\mathcal{M}}^{\top }{\Sigma }_{1n}{X}_{\mathcal{M}}+I_{\left|\mathcal{M}\right|}/{\sigma }_{1n}^2\right)}^{-1}{X}_{\mathcal{M}}^{\top }{\Sigma }_{1n}\right\}Y,$$

$$T_{\mathcal{M}}={\sigma }_{1n}^{-\left|\mathcal{M}\right|}{\left|X_{\mathcal{M}}^{\top }{\Sigma }_{1n}{X}_{\mathcal{M}}+I_{\left|\mathcal{M}\right|}/{\sigma }_{1n}^2\right|}^{-1},$$

and ${\Sigma }_{1n}=I_n-{\left({\Sigma }_{0n}^{+}+I_n\right)}^{-1}$, Σ_0n is the covariance matrix for the prior of f(U).

Furthermore, define the likelihood ratio between model $\mathcal{M}$ and the true model $\mathcal{A}$ as $\text{PR}(\mathcal{M},\mathcal{A})$. If Conditions A, C hold, $\text{PR}(\mathcal{M},\mathcal{A})$ is bounded by

$$PR(\mathcal{M},\mathcal{A})=\frac{\text{Pr}(\gamma =\mathcal{M}\mid Y,\sigma )}{\text{Pr}(\gamma =\mathcal{A}\mid Y,\sigma )}⪯p_n^{-1.5\delta (\left|\mathcal{M}\right|-\left|\mathcal{A}\right|)+0.5\kappa }\text{exp}\left\{-\frac{1}{2{\sigma }^2}\left(R_{\mathcal{M}}-R_{\mathcal{A}}\right)\right\}.$$ (4)

Remark 2. Lemma 1 gives the explicit form of the posterior probability for any given model $\mathcal{M}$ and puts an upper bound on the likelihood ratio between model $\mathcal{M}$ and the true model $\mathcal{A}$. Intuitively from (4), when ${\sigma }_{1n}^2$ is sufficiently large, $R_{\mathcal{M}}$ is close to

$$R_{\mathcal{M}}^{*}=Y^{\top }\left\{{\Sigma }_{1n}-{\Sigma }_{1n}{X}_{\mathcal{M}}{\left(X_{\mathcal{M}}^{\top }{\Sigma }_{1n}{X}_{\mathcal{M}}\right)}^{-1}{X}_{\mathcal{M}}^{\top }{\Sigma }_{1n}\right\}Y.$$

So if Σ_1n is taken to be CD^⊤D, which means Σ_0n = {(I_n – CD^⊤D)⁻¹ − I_n}⁺, then $R_{\mathcal{M}}^{*}$ is proportional to the sum of squared residuals under model $\mathcal{M}$, after applying the difference-based method. It could be interpreted as the goodness of fit. Additionally, the first term $p_n^{-1.5\delta (\left|\mathcal{M}\right|-\left|\mathcal{A}\right|)}$ in (4) could be regarded as the penalty on the model size. So it is mostly analogous to a L₀ penalized method. As σ_1n diverges fast, we can directly work with $R_{\mathcal{A}}^{*}$ and $R_{\mathcal{M}}^{*}$ instead of $R_{\mathcal{M}}-R_{\mathcal{A}}$. The following two lemmas present some properties of $R_{\mathcal{A}}^{*}$ and $R_{\mathcal{M}}^{*}$.

Lemma 2. For any model $\mathcal{M}$ containing the true model, i.e $\mathcal{A}\subseteq \mathcal{M}$, if conditions A, E, F hold, then

$$R_{\mathcal{M}}^{*}~C{\sigma }^2{\chi }_{n-m-\left|\mathcal{M}\right|}^{2}a.s.,$$

$$R_{\mathcal{A}}^{*}-R_{\mathcal{M}}^{*}~C{\sigma }^2{\chi }_{\left|\mathcal{M}\right|-\left|\mathcal{A}\right|}^{2}a.s\mathrm{..}$$

Lemma 3. Suppose that Conditions A, C, D are satisfied, then for any g_n → ∞ and ϵ > 0,

1. $\text{Pr}\left(R_{\mathcal{A}}-R_{\mathcal{A}}^{*}>g_n\right)\le \text{exp}\left(-c^{\prime }n{\lambda }_1{\sigma }_{1n}^2{g}_n\right)$, for some c′ > 0, where λ₁ is defined in Condition C.

2. $\text{Pr}\left(\left|R_{\mathcal{A}}^{*}/\left(Cn{\sigma }^2\right)-1\right|>ϵ\right)\le \text{exp}(-cn)$, for some c > 0.

Remark 3. Lemma 2 shows for over-fitted models, after applying the difference operation, the sum of squared residuals has an asymptotic χ² distribution with the degrees of freedom as $n-m-\left|\mathcal{M}\right|$. It also gives the asymptotic distribution of $R_{\mathcal{A}}^{*}-R_{\mathcal{M}}^{*}$. The difference of $R_{\mathcal{A}}$ and $R_{\mathcal{A}}^{*}$ is further bounded in Lemma 3. Lemma 3 (ii) is straightforward by using the tail bound with χ² distributions.

Theorem 4. (Strong Selection Consistency under fixed σ²). Suppose that Conditions A, B, C, D, E, F hold for the partially linear model in (1) with ln p_n = o(n), and $\left|\mathcal{A}\right|=o\left(m_n\right)$ are valid, we have

$$\text{Pr}\left(\gamma =\mathcal{A}\left|Y,{\sigma }^2,\right|\gamma \left|\le m_n+\right|\mathcal{A}\mid \right)\stackrel{P}{\to }1\text{ as }n\to \infty ,$$

where m_n = cn/ ln p_n, as defined in Condition D.

Remark 4. It suffices to show

$$\sum _{\mathcal{M}\ne \mathcal{A},\left|\mathcal{M}\right|\le m_n+\left|\mathcal{A}\right|}\text{PR}(\mathcal{M},\mathcal{A})=\sum _{\mathcal{M}\ne \mathcal{A},\left|\mathcal{M}\right|\le m_n+\left|\mathcal{A}\right|}\frac{\text{Pr}(\gamma =\mathcal{M}\mid Y,\sigma )}{\text{Pr}(\gamma =\mathcal{A}\mid Y,\sigma )}\stackrel{P}{\to }0.$$

Recall that, by (4) in Lemma 1, we have $PR(\mathcal{M},\mathcal{A})⪯p_n^{-1.5\delta (\left|\mathcal{M}\right|-\left|\mathcal{A}\right|)+0.5\kappa }\text{exp}\left\{-\frac{1}{2{\sigma }^2}\left(R_{\mathcal{M}}-R_{\mathcal{A}}\right)\right\}$. Inspired by [21], we first divide the model space into 3 disjoint parts, ${\mathcal{P}}_i,i=1,2,3$ defined as below. In each group, we prove the sum of likelihood ratio converges to 0 in probability.

1. Consider the set of overfitted models ${\mathcal{P}}_1=\left\{\mathcal{M}:\mathcal{A}\subseteq \mathcal{M},|\mathcal{M}|\le m_n+|\mathcal{A}|\right\}$. Model $\mathcal{M}$ in this group contains all active variables, so $R_{\mathcal{A}}-R_{\mathcal{M}}$ might be large. However since $\left|\mathcal{M}\right|-\left|\mathcal{A}\right|>0$, the number of extra spurious variables is penalized at the rate of $p_n^{-c}$.

2. For large and including some inactive variables models ${\mathcal{P}}_2=\left\{\mathcal{M}:\mathcal{A}⊈\mathcal{M},K\left|\mathcal{A}\right|<\left|\mathcal{M}\right|\le m_n+\left|\mathcal{A}\right|\right\}$, let $\mathcal{M}\vee \mathcal{A}$ be the union of model $\mathcal{M}$ and $\mathcal{A}$, then any model $\mathcal{M}$ in this group has $\mathcal{M}\vee \mathcal{A}\in {\mathcal{P}}_1$. Although size of $\mathcal{M}\vee \mathcal{A}$ may exceed $m_n+\left|\mathcal{A}\right|$, but since m_n dominates $\left|\mathcal{A}\right|$ so it is reasonable to assume the difference is negligible. Model $\mathcal{M}\vee \mathcal{A}$ will also have a better fit than $\mathcal{M}$. Thus we can control $\text{exp}\left\{-\frac{1}{2{\sigma }^2}\left(R_{\mathcal{M}}-R_{\mathcal{A}}\right)\right\}$ by bounding the value of $\text{exp}\left\{-\frac{1}{2{\sigma }^2}\left(R_{\mathcal{M}\vee \mathcal{A}}-R_{\mathcal{A}}\right)\right\}$. Since the size of models in ${\mathcal{P}}_2$ is large, the growth of $\text{exp}\left\{-\frac{1}{2{\sigma }^2}\left(R_{\mathcal{M}}-R_{\mathcal{A}}\right)\right\}$ is under control.

3. Now consider the set of underfitted models missing some active variables, which is formulated as ${\mathcal{P}}_3=\left\{\mathcal{M}:\mathcal{A}⊈\mathcal{M},\left|\mathcal{M}\right|\le K\left|\mathcal{A}\right|\right\}$, where K is the constant defined in Condition B. For any model $\mathcal{M}$ in this group, at least one active variable is missed. By Condition B on the identifiability of the active variables, $R_{\mathcal{M}}-R_{\mathcal{A}}$ will be large since model $\mathcal{M}$ does not have a good fit, thus the value of $PR(\mathcal{M},\mathcal{A})$ is controlled.

Remark 5. In this paper, we only consider models that are not unreasonably large, that is, $\left|\mathcal{M}\right|\le m_n+\left|\mathcal{A}\right|$, where m_n is at the order of n/ ln p_n. There is a reason behind the choice of m_n. It can be shown that the marginal probability for the inclusion of a single variable i.e. q = Pr(γ_i = 1) somehow controls the size of the selected model. Let $\widehat{\mathcal{M}}=\underset{\mathcal{M}}{\text{argmax}}\text{Pr}\left(\gamma =\mathcal{M}\mid Y,{\sigma }^2\right)$. For any fixed choice of q, we can derive the upper bound on the selected model size

$$\left|\widehat{\mathcal{M}}\right|⪯\left|\mathcal{A}\right|+\frac{c\left|\mathcal{A}\right|\text{ln\hspace{0.17em}}{p}_n}{-\text{ln\hspace{0.17em}}q}+\frac{c^{\prime }n}{-\text{ln\hspace{0.17em}}q}.$$

Since we require $q_n~p_n^{-1}$ and $\left|\mathcal{A}\right|=o\left(\frac{n}{\text{ln\hspace{0.17em}}{p}_n}\right)$ in Conditions A, D, it reduces to

$$\left|\widehat{\mathcal{M}}\right|=O\left(\frac{n}{\text{ln\hspace{0.17em}}{p}_n}\right).$$

Theorem 5. (Strong Selection Consistency under unknown σ²). Suppose that Condition A, B, C, D, E, F hold for the partially linear model in (1) with ln p_n = o(n), and $\left|\mathcal{A}\right|=o\left(m_n\right)$ are valid, we have

$$\text{Pr}\left(\gamma =\mathcal{A}|Y,|\gamma \left|\le m_n+\right|\mathcal{A}\mid \right)\stackrel{P}{\to }1\text{ as }n\to \infty ,$$

where m_n = cn/ ln p_n, as defined in Condition D.

By further integrating out σ² and applying some inequalities, the problem reduces to intermediate steps in Theorem 4. Please refer to Section 4 for the proof.

Remark 6. As linear models are special cases of partially linear models, the proposed BSM-DP variable selection method may be directly applicable to linear models. We have also studied theoretical properties, finite sample performance and computational time of the BSM-DP under the setting of linear models and compared with BASAD [21] and SG [22]. To save space, all material related to BSM-DP for linear models are put in the supplementary material.

3. Numerical study

3.1. Simulation study

3.1.1. Simulation settings and the choice of hyperparameters

In this section, we compare the performance of the proposed method with several other existing methods including penalized methods on partial residuals and methods based on partial correlation of partial residuals. The penalized methods include the famous LASSO [25] and SCAD [8] tuned by BIC. The R packages msgps, ncvreg are used for LASSO and SCAD. Methods based on partial correlation include PC-simple algorithm on the partial residuals (PC-PR) [3] and threshold partial correlation on partial residuals (TPC-PR) [20]. Both PC-PR and TPC-PR select variables based on the magnitude of the partial correlation between the partial residuals of the response and the corresponding predictors, while the difference is on the threshold used for partial correlations. TPC-PR uses a threshold depending on the kurtosis of X, so the normality assumption for X is not necessary. For TPC-PR, we also consider a fine tuning on the critical value cT(α, n, $\widehat{\kappa }$, m), where c is the tuning parameter chosen by EBIC [5]. The method is denoted as TPC-PR.EBIC.

First we need to specify the hyper-parameters σ₀, σ_1n, q_n, m, α₀, β₀. Partially refer to the choice in [21], we use α₀ = 2, β₀ = 5, and σ₀ = 0.1 for our proposed method (BSM-DP). The order of the difference operator is set to be m = ⌊5n^1/5⌋. Additionally, the variance for the diffusing prior is set as ${\sigma }_{1n}=\sqrt{\text{max}\left\{\frac{p_n^{2.1}}{100n},\text{ln}(n)\right\}}$, and we choose q_n = Pr(γ_i = 1) such that $\text{Pr}\left({\sum }_{j=1}^{p_n}{\gamma }_j>K\right)=0.1$, for a prespecified value of K. The value of K can be our preliminary guess for the size of active set, and for example, we can use the size of active set selected by LASSO. In this paper we simply set K = 10. In each of the following case, we allow 6000 iterations, and treat the first 3000 as burn-in samples. We report simulation results based on both MAP and MPM for our proposed method.

We fix n = 200, p = 1000 and the true active set $\mathcal{A}=(1,2,5,8)$ with coefficients of ${\beta }_{\mathcal{A}}=(1.5,2.0,2.5,3.0)$. The error ϵ_i is drawn from standard normal distribution $\mathcal{N}(0,1)$. Fixed design of U_i = i/n, i ∈ {1, …, n} is used with three different types of X:

• Case 1. Type I normal distribution with autoregressive covariance matrix: $X~\mathcal{N}(0,\Sigma )$, where Σ_ij = ρ^|i−j|.

• Case 2. Type II normal distribution with compound symmetric covariance matrix: $X~\mathcal{N}(0,\Sigma )$, where Σ_ij = 1 for i = j and Σ_ij = ρ for i ≠ j.

• Case 3. Type III mixture of normals: X is sampled from $\mathcal{N}(0,\Sigma )$ with probability 0.9 and from $\mathcal{N}(0,9\Sigma )$ with probability 0.1, where Σ is the compound symmetric correlation matrix with correlation ρ.

In each case, we also consider low and high correlations at ρ = 0.2 and ρ = 0.8 separately, with two choices of the nonparametric component f(U) = U² and f(U) = sin(2πU). The following evaluation criteria are used for comparing methods based on 500 replications:

• $p_{{\mathcal{A}}^c}^{\text{max}}$ and $p_{\mathcal{A}}^{\text{min}}$: the average for the maximal of marginal posterior probabilities on true inactive covariates, and the average for the minimal of marginal posterior probabilities on true active covariates.

• $p_{\mathcal{A}=\mathcal{M}}$: the proportion of replications with the exact model being selected.

• $p_{\mathcal{A}\in \mathcal{M}}$: the proportion of replications with all true active variables being selected.

• p_i: the proportion of replications that the ith true active variable is selected successfully, i ∈ {1, 4}.

• TP (true positive): the average number of true active variables selected.

• FP (false positive): the average number of selected variables that are actually inactive.

• ME (model error): ${(\widehat{\beta }-\beta )}^{\top }\text{cov}(X)(\widehat{\beta }-\beta )$.

Note that for those existing methods, partial residuals are firstly obtained. Getting the partial residuals involves the value of E(Y|U), which is estimated by the local linear regression, followed by [20]. The bandwidth is chosen via plug-in method using R package KernelSmooth. Afterwards, with LASSO and SCAD, we are able to estimate β and achieve variable selection simultaneously in the second step. While for the partial correlation methods including PC and TPC, an estimation of the active set $\widehat{\mathcal{A}}$ needs to be obtained first, then $\widehat{\beta }$ is estimated by regressing the partial residuals on $\widehat{\mathcal{A}}$ through the least squares method. Details can be found in [20]. In this simulation, we use the posterior mean of β as $\widehat{\beta }$ for the proposed method.

3.1.2. Simulation results

Table 1 – Table 3 record the mean results from those 500 replications. Case 1 is with the decayed autoregressive covariance matrix. Under the case with low correlation ρ = 0.2, all methods perform well regardless of the type of the nonparametric function. For the situation with high correlation of ρ = 0.8, LASSO is prone to overfit the model, with the exact model being selected only around 20% of the time.

Table 1.

Summarized simulation results for Case 1: p = 1000, n = 200 and X is sampled from normal distribution with autoregressive correlation matrix. The reported values are means of different performance measures averaged over 500 replications. The methods compared include LASSO and SCAD on partial residuals tuned by BIC (LASSO.BIC, SCAD.BIC), PC-simple algorithm on the partial residuals (PC-PR), threshold partial correlation on partial residuals (TPC-PR, TPC-PR.EBIC), proposed method with model selected by MAP and MPM (BSM-DP.MAP, BSM-DP.MPM). The details of different methods and measures are provided in Section 3.1.1.

	Method	$p_{{\mathcal{A}}^c}^{\text{max}}$	$p_{\mathcal{A}}^{\text{min}}$	$p_{\mathcal{A}=\mathcal{M}}$	$p_{\mathcal{A}\in \mathcal{M}}$	p1	p4	TP	FP	ME
	LASSO.BIC			0.760	1.000	1.000	1.000	4.000	0.256	0.313
	SCAD.BIC			0.932	1.000	1.000	1.000	4.000	0.082	0.026
f(u) = u2	PC-PR			0.966	0.994	0.994	1.000	3.994	0.030	0.040
ρ = 0.2	TPC-PR			0.964	0.994	0.994	1.000	3.994	0.032	0.041
	TPC-PR.EBIC			0.994	1.000	1.000	1.000	4.000	0.010	0.027
	BSM-DP.MAP (new)	0.073	1.000	0.974	1.000	1.000	1.000	4.000	0.032	0.026
	BSM-DP.MPM (new)			0.974	1.000	1.000	1.000	4.000	0.030	0.026
	LASSO.BIC			0.738	1.000	1.000	1.000	4.000	0.288	0.334
	SCAD.BIC			0.924	1.000	1.000	1.000	4.000	0.120	0.029
f(u) = sin(2πu)	PC-PR			0.996	0.994	0.994	1.000	3.994	0.032	0.045
ρ = 0.2	TPC-PR			0.958	0.994	0.994	1.000	3.994	0.040	0.045
	TPC-PR.EBIC			0.982	1.000	1.000	1.000	4.000	0.018	0.029
	BSM-DP.MAP (new)	0.074	1.000	0.974	1.000	1.000	1.000	4.000	0.028	0.026
	BSM-DP.MPM (new)			0.970	1.000	1.000	1.000	4.000	0.032	0.026
	LASSO.BIC			0.222	1.000	1.000	1.000	4.000	1.266	0.309
	SCAD.BIC			0.984	0.998	0.998	1.000	3.998	0.020	0.032
f(u) = u2	PC-PR			0.644	0.652	0.768	1.000	3.652	0.094	0.344
ρ = 0.8	TPC-PR			0.652	0.660	0.774	1.000	3.660	0.092	0.338
	TPC-PR.EBIC			0.846	0.858	0.886	1.000	3.858	0.038	0.149
	BSM-DP.MAP (new)	0.053	1.000	0.990	1.000	1.000	1.000	4.000	0.010	0.023
	BSM-DP.MPM (new)			0.994	1.000	1.000	1.000	4.000	0.006	0.023
	LASSO.BIC			0.172	1.000	1.000	1.000	4.000	1.362	0.321
	SCAD.BIC			0.986	0.998	0.998	1.000	3.998	0.014	0.040
f(u) = sin(2πu)	PC-PR			0.650	0.656	0.796	1.000	3.998	0.014	0.348
ρ = 0.8	TPC-PR			0.662	0.668	0.804	1.000	3.666	0.132	0.337
	TPC-PR.EBIC			0.840	0.854	0.906	1.000	3.854	0.064	0.159
	BSM-DP.MAP (new)	0.055	1.000	0.982	1.000	1.000	1.000	4.000	0.018	0.023
	BSM-DP.MPM (new)			0.984	1.000	1.000	1.000	4.000	0.016	0.023

Table 3.

Summarized simulation results for Case 3: p = 1000, n = 200 and X is sampled from mixture of normals with compound symmetric matrix. The reported values are means of different performance measures averaged over 500 replications. The methods compared include LASSO and SCAD on partial residuals tuned by BIC (LASSO.BIC, SCAD.BIC), PC-simple algorithm on the partial residuals (PC-PR), threshold partial correlation on partial residuals (TPC-PR, TPC-PR.EBIC), proposed method with model selected by MAP and MPM (BSM-DP.MAP, BSM-DP.MPM). The details of different methods and measures are provided in Section 3.1.1. Results under high correlation ρ = 0.8 are highlighted.

	Method	$p_{{\mathcal{A}}^c}^{\text{max}}$	$p_{\mathcal{A}}^{\text{min}}$	$p_{\mathcal{A}=\mathcal{M}}$	$p_{\mathcal{A}\in \mathcal{M}}$	p1	p4	TP	FP	ME
	LASSO.BIC			0.156	1.000	1.000	1.000	4.000	2.434	0.475
	SCAD.BIC			0.994	1.000	1.000	1.000	4.000	0.008	0.022
f(u) = u2	PC-PR			0.476	0.690	0.724	1.000	3.676	1.304	0.927
ρ = 0.2	TPC-PR			0.574	0.574	0.636	1.000	3.552	0.622	1.252
	TPC-PR.EBIC			0.680	0.694	0.726	1.000	3.680	0.612	0.891
	BSM-DP.MAP (new)	0.046	1.000	0.984	1.000	1.000	1.000	4.000	0.016	0.015
	BSM-DP.MPM (new)			0.986	1.000	1.000	1.000	4.000	0.014	0.015
	LASSO.BIC			0.228	1.000	1.000	1.000	4.000	2.194	0.449
	SCAD.BIC			0.990	1.000	1.000	1.000	4.000	0.012	0.018
f(u) = sin(2πu)	PC-PR			0.572	0.768	0.802	1.000	3.776	0.982	0.670
ρ = 0.2	TPC-PR			0.618	0.622	0.682	1.000	3.602	0.544	1.125
	TPC-PR.EBIC			0.748	0.768	0.804	1.000	3.764	0.438	0.667
	BSM-DP.MAP (new)	0.050	1.000	0.986	1.000	1.000	1.000	4.000	0.016	0.014
	BSM-DP.MPM (new)			0.988	1.000	1.000	1.000	4.000	0.012	0.014
	LASSO.BIC			0.000	0.998	0.998	1.000	3.998	14.060	0.395
	SCAD.BIC			0.380	0.382	0.432	1.000	3.374	0.120	0.652
f(u) = u2	PC-PR			***	***	***	***	***	***	***
ρ = 0.8	TPC-PR			***	***	***	***	***	***	***
	TPC-PR.EBIC			***	***	***	***	***	***	***
	BSM-DP.MAP (new)	0.092	1.000	0.962	1.000	1.000	1.000	4.000	0.042	0.025
	BSM-DP.MPM (new)			0.964	1.000	1.000	1.000	4.000	0.040	0.025
	LASSO.BIC			0.000	0.996	0.996	1.000	3.996	14.002	0.418
	SCAD.BIC			0.422	0.426	0.492	1.000	3.416	0.022	0.672
f(u) = sin(2πu)	PC-PR			***	***	***	***	***	***	***
ρ = 0.8	TPC-PR			***	***	***	***	***	***	***
	TPC-PR.EBIC			***	***	***	***	***	***	***
	BSM-DP.MAP (new)	0.082	1.000	0.960	1.000	1.000	1.000	4.000	0.042	0.020
	BSM-DP.MPM (new)			0.970	1.000	1.000	1.000	4.000	0.032	0.020

^***stands for the cases when a single replication takes more than 48 hours. So 500 replications can not be done in timely manner.

It gets more challenging to identify true covariates under dense correlation, which is Case 2 with the compound symmetric covariance matrix. The exact-fit rates are much lower for most of the methods as compared to that in Case 1. It is noteworthy that under high correlation ρ = 0.8, except for our proposed BSM-DP, most other methods do not perform well. LASSO consistently selects a larger model with about 16 spurious covariates on average, while SCAD tuned by BIC is prone to select a smaller model. PC and TPC work evidently slow under dense correlation, when ρ = 0.2, it takes around 12 hours for PC, TPC and TPC-EBIC to finish one replication. More than 48 hours are needed when ρ = 0.8, thus we mark it as stars (***) since 500 replications can not be done in timely manner. Our proposed method (BSM-DP) gives the best results, with high exact-fit rates (above 95%) even under the high dense correlation situation.

In Case 3, X is generated from a mixture normal distribution, with a heavier tail than the normal distribution. Since PC-PR relies heavily on the normality of the covariates, it gives poor results. The updated version of TPC without assuming normality shows improvement. Our proposed method (BSM-DP) still stands out in the comparison with around 95% perfect exact-fit rates.

Overall, when correlation increases, LASSO tuned by BIC tends to overfit the model while SCAD tuned by BIC is more likely to select a smaller model. When X is normally distributed, PC and TPC are similar. But when the normality assumption is violated, TPC performs better than PC. The newly proposed method BSM-DP performs consistently the best, regardless of the correlation strength and distribution of X. The exact-fit rates for all cases are all above 90%.

Remark 7. (On model selection procedure) In our simulation study, models selected by MAP and MPM are very similar to each other. We do not need to select the threshold with MAP. With the MPM, the jth variable is selected if its posterior probability Pr(γ_j = 1|Y) ≥ 0.5. In order to investigate the impact of different thresholds other than 0.5, we further explore one simulation setting Case 2 with ρ = 0.8, and consider various threshold values from 0 to 1. The results are presented in Fig. 2.. With a smaller threshold, more spurious variables are likely to enter the model, so the false discovery rate (dotted red line) is higher. While with a larger threshold which associates with a more stringent selection criterion, we have a higher chance to miss active variables. It is worth noting that although for our simulation setting with ${\beta }_{\mathcal{A}}=(1.5,2.0,2.5,3.0)$ shown in Fig. 2.(a), the true positive rate (dashed green line) is consistently high as all active variables have marginal inclusion probabilities as 1, but generally we may expect a drop when threshold approaching 1 for most cases as in Fig. 2.(b) with lower signal ${\beta }_{\mathcal{A}}=(0.8,1.3,1.8,2.3)$. The exact-fit rate (solid blue line), which gives the proportion of replications with exact model being selected, is reasonably good, as long as the threshold is neither too large nor too small. Overall, 0.5 looks like a good choice.

Fig. 2.

Impact of model selection performance using different threshold values for the posterior probability. The criteria displayed to measure performance include exact-fit rate (solid blue line), false discovery rate (dotted red line) and true positive rate (dashed green line). A range of threshold values from 0 and 1 are used to plot the curve for each criteria. Two signal values with different strengths are considered: ${\beta }_{\mathcal{A}}=(1.5,2.0,2.5,3.0)$ and ${\beta }_{\mathcal{A}}=(0.8,1.3,1.8,2.3)$.

We also compare the computation time required by different methods. Among the five methods mentioned above, LASSO and SCAD are the fastest which take about 2 minutes in R to finish estimation for one replication. PC and TPC are fast as well when covariate X has decayed covariance matrix. They become dramatically slow with high and dense correlation. More than 48 hours is needed for PC using R to finish one replication when the covariance of X is compound symmetric with ρ = 0.8. It may get worse with higher dimensional covariates and larger active groups, since the computational time for both PC and TPC grows polynomially with them. Time is recorded based on Macbook Pro early 2015 with 2.7 GHZ, Intel i5 and 8 GB.

On the contrary, the computation burden for the newly proposed Bayesian method (BSM-DP) is moderate. Among all simulation settings, the slowest one takes about 12 minutes to finish 6000 iterations for one replication using Julia 0.6. As discussed in Remark 1 Section 2.1, the Bayesian subset modeling is scalable, and the computational time only grows approximately linearly with the dimension of covariates. Based on the estimation procedure, the computational complexity for each iteration is n(p ∨ |γ|² ∨ n²), where |γ| is the current active model size. To explore the change in the computation time for different (and especially higher) dimensions of covariates, we record the CPU time to finish 6000 iterations with various p from 100 to 4000, in the simulation setting Case 1 ρ = 0.8. The result is presented in Fig. 3.. The computation time increases nearly linearly with the dimension p. It is not perfectly linear since the number of iterations until convergence seems to grow with p. We notice that for small p (e.g. p < 500), it usually only takes a few iterations to converge and end up with a small |γ|, but it often requires more iterations when p gets larger. There is one place in the above plot which shows a large jump. This might relate to the caching limit on the hardware, especially when p is large.

Fig. 3.

Change in computational time (in minutes) when the dimension of covariates increases from 100 to 4000. The CPU time is estimated by the median computation time consumed among 10 replications for each dimension setting.

3.2. A real data example — supermarket data analysis

In this section, the proposed method is applied to analyze a supermarket data set mentioned in [6, 20, 26]. The data set contains n = 464 daily records of the number of customers, which is the response variable and the sales of p = 6398 products which are predictors. Both the response variable and the predictors are standardized to have zero mean and unit variance. The plot on the left of Fig. 4 shows the relationship between the number of customers and the days. The periodicity of Y is obvious, thus it is reasonable to model Y with a PLM, which takes time variation into account. A covariate U_i = i/n is introduced to represent time. To check the correlation among predictors, we plot the histogram of the sampled correlation in Fig. 4., which shows some moderate correlation. We randomly select 75% of the observations (X_i, Y_i, U_i), i ∈ {1,…, 464) as the training set and keep the remaining 25% as the testing set. The PLM is fitted and variables are selected using the training dataset, then the mean-squared errors of the selected model on the testing dataset are calculated to evaluate the model fit. This procedure is repeated 100 times, and Table 4 summarizes the average size of the selected models and the mean squared errors.

Fig. 4.

Visualizations to display features of the (standardized) supermarket data set. The plot on the left gives the trend of daily number of customers entering a supermarket for 464 days. The histogram on the right describes the distribution of the correlation among sampled predictors.

Table 4.

Comparions of the resulting model size and the mean squared errors by different methods. The values in the table are the means and the corresponding standard errors (in the parenthesis) over the 100 replications. The methods compared are LASSO and SCAD on partial residuals tuned by BIC (SIS-LASSO.BIC, SIS-SCAD.BIC), PC-simple algorithm on the partial residuals (SIS-PC-PR), threshold partial correlation on partial residuals (SIS-TPC-PR, SIS-TPC-PR.EBIC), proposed method with model selected by MPM tuned by EBIC (SIS-BSM-DP.EBIC).

Method	Model Size (s.e.)	MSE on training set (s.e.)	MSE on testing set (s.e.)
SIS-LASSO.BIC	28.80 (3.32)	0.0575 (0.0030)	0.0836 (0.0120)
SIS-SCAD.BIC	15.75 (5.44)	0.0647 (0.0062)	0.0907 (0.0143)
SIS-PC-PR	12.94(1.03)	0.0497 (0.0031)	0.0847 (0.0112)
SIS-TPC-PR	9.81 (0.92)	0.0540 (0.0034)	0.0860 (0.0109)
SIS-TPC-PR.EBIC	8.50 (0.98)	0.0559 (0.0038)	0.0867 (0.0117)
SIS-BSM-DP.EBIC (new)	7.95(1.91)	0.0610 (0.0055)	0.0864 (0.0122)

To take the dimension down to a moderate scale, we first apply the SIS [9] on the partial residual to only keep the top 2000 predictors as the set subjected to variable selection. We also implement the LASSO.BIC, SCAD.BIC, PC-PR, TPC-PR and TPC-PR-EBIC on the same data set as comparisons. The choice of the hyperparameters for BSM-DP is the same as the set up in the simulation. we complete 10000 iterations, with the first 6000 as burn-in samples, and rank covariates by their marginal posterior probabilities Pr(γ_j = 1|Y). The candidate set is further selected by EBIC.

After obtaining partial residuals, with LASSO and SCAD, we are able to obtain $\widehat{\beta }$ and select variables simultaneously. But for PC, TPC, and BSM-DP, an estimation of the active set $\widehat{\mathcal{A}}$ is first obtained, then $\widehat{\beta }$ is estimated by regressing the partial residuals on $\widehat{\mathcal{A}}$ through OLS. Since we derive the theoretical property of γ, we only use BSM-DP to select the set of active covariates, $\widehat{\beta }$ is also obtained by regressing the partial residuals on the selected covariate set.

As shown in Table 4, LASSO gives the most conservative result with an average size of selected models as 28.80. On the other hand, the MSE is much smaller on the training than that on the testing. This suggests that the LASSO may be overfitting. SCAD selects smaller models with size 15.75 on average, but having large error on testing set. PC, TPC, TPC.EBIC and the newly proposed BSM-DP all select even smaller models with the average size less than 10. Among all, our proposed BSM-DP selects the smallest number of covariates with very similar value of the mean squared error on the testing data set. Fig. 5. illustrates a comparison on the estimated nonparametric function obtained by different methods.

Fig. 5.

The estimates of the nonparametric function for the supermarket data set by different methods including: LASSO and SCAD on partial residuals tuned by BIC (SIS-LASSO.BIC, SIS-SCAD.BIC), PC-simple algorithm on the partial residuals (SIS-PC-PR), threshold partial correlation on partial residuals (SIS-TPC-PR, SIS-TPC-PR.EBIC), and proposed method with model selected by MPM tuned by EBIC (SIS-BSM-DP.EBIC).

4. Technical proofs

This section includes technical proofs for Lemma 1 – Lemma 3 and Theorem 4 – Theorem 5.

Proof of Lemma 1. Note that, as q_n → 0, which is stated in Condition A, we first write out the posterior distribution for parameters as

$$\text{Pr}(\mathit{\gamma }=\mathcal{M},\beta ,\sigma ,\alpha \mid Y)\propto \pi (Y\mid \alpha ,\gamma ,\beta ,\sigma )\pi (\alpha \mid \sigma )\pi (\beta \mid \gamma ,\sigma )\pi (\gamma )\pi \left({\sigma }^2\right)\propto \text{exp}\left\{-\frac{1}{2{\sigma }^2}{\left(Y-\alpha -X_{\mathcal{M}}{\beta }_{\mathcal{M}}\right)}^{\top }\left(Y-\alpha -X_{\mathcal{M}}{\beta }_{\mathcal{M}}\right)-\frac{1}{2{\sigma }^2}{\alpha }^{\top }{\Sigma }_{0n}^{+}\alpha \right\}{q}_n^{\left|\mathcal{M}\right|}{\sigma }^{-2n-p_n}{\sigma }_{1n}^{-\left|\mathcal{M}\right|}{\sigma }_0^{-\left(p_n-\left|\mathcal{M}\right|\right)}\text{exp}\left\{-\frac{1}{2{\sigma }_{1n}^2{\sigma }^2}{\beta }_{\mathcal{M}}^{\top }{\beta }_{\mathcal{M}}-\frac{1}{2{\sigma }_0^2{\sigma }^2}{\beta }_{{\mathcal{M}}^c}^{\top }{\beta }_{{\mathcal{M}}^c}\right\}\pi \left({\sigma }^2\right).$$

We first integrate out α, and it follows that

$$\text{Pr}(\mathit{\gamma }=\mathcal{M},\beta ,\sigma \mid Y)\propto q_n^{\left|\mathcal{M}\right|}{\sigma }^{-n-p_n}{\sigma }_{1n}^{-\left|\mathcal{M}\right|}{\sigma }_0^{-\left(p_n-\left|\mathcal{M}\right|\right)}\text{exp}\left\{-\frac{1}{2{\sigma }_{1n}^2{\sigma }^2}{\beta }_{\mathcal{M}}^{\top }{\beta }_{\mathcal{M}}-\frac{1}{2{\sigma }_0^2{\sigma }^2}{\beta }_{{\mathcal{M}}^c}^{\top }{\beta }_{{\mathcal{M}}^c}\right\}\text{exp}\left[-\frac{1}{2{\sigma }^2}{\left(Y-X_{\mathcal{M}}{\beta }_{\mathcal{M}}\right)}^{\top }\left\{I_n-{\left({\Sigma }_{0n}^{+}+I_n\right)}^{-1}\right\}\left(Y-X_{\mathcal{M}}{\beta }_{\mathcal{M}}\right)\right][{\Sigma }_{0n}^{+}+{I_n|}^{-1/2}\pi \left({\sigma }^2\right).$$

Denote ${\Sigma }_{1n}=I_n-{\left({\Sigma }_{0n}^{+}+I_n\right)}^{-1}$. It follows by integrating out β that

$$\text{Pr}(\gamma =\mathcal{M},\sigma \mid Y)\propto q_n^{\left|\mathcal{M}\right|}{\sigma }^{-n}{\sigma }_{1n}^{-\left|\mathcal{M}\right|}{\left|{\Sigma }_{0n}^{+}+I_n\right|}^{-1/2}{\left|X_{\mathcal{M}}^{\top }{\Sigma }_{1n}{X}_{\mathcal{M}}+I_{\left|\mathcal{M}\right|}/{\sigma }_{1n}^2\right|}^{-1/2}\text{exp}\left(-\frac{1}{2{\sigma }^2}{R}_{\mathcal{M}}\right)\pi \left({\sigma }^2\right),$$ (5)

where

$$R_{\mathcal{M}}=Y^{\top }\left\{{\Sigma }_{1n}-{\Sigma }_{1n}{X}_{\mathcal{M}}{\left(X_{\mathcal{M}}^{\top }{\Sigma }_{1n}{X}_{\mathcal{M}}+I_{\left|\mathcal{M}\right|}/{\sigma }_{1n}^2\right)}^{-1}{X}_{\mathcal{M}}^{\top }{\Sigma }_{1n}\right\}Y.$$

Let X* = C^1/2DX, ${\lambda }_1^{*}={\text{min}}_{\mathcal{M}:\left|\mathcal{M}\right|\le m_n+\left|\mathcal{A}\right|}{\lambda }_{\text{min}}\left(\frac{1}{n-m}{X}_{\mathcal{M}}^{*\top }{X}_{\mathcal{M}}^{*}\right)$, ${\lambda }_2^{*}={\text{max}}_{\mathcal{M}:\mathcal{M}\subseteq \mathcal{A}}{\lambda }_{\text{max}}\left(\frac{1}{n-m}{X}_{\mathcal{M}}^{*\top }{X}_{\mathcal{M}}^{*}\right)$. As $D^{\top }D\to \left(\begin{array}{cc}{0}_{m\times m}& 0_{m\times (n-m)}\\ 0_{(n-m)\times m}& I_{n-m}\end{array}\right)$, for any model $\mathcal{M}$, $X_{\mathcal{M}}^{*\top }{X}_{\mathcal{M}}^{*}$ is asymptotically equal to $C{X}_{\mathcal{M}}^{\top }{X}_{{\mathcal{M}}^{\prime }}$, where ${\mathcal{M}}^{\prime }$ is the subset of $\mathcal{M}$ by taking last n − m elements. By Conditions C, E, it can be shown that ${\lambda }_1^{*}~{\lambda }_1$ and ${\lambda }_2^{*}~{\lambda }_2$. Now we put bound on $T_{\mathcal{M}}/T_{\mathcal{A}}$. For any model $\mathcal{M}$:

$$\frac{T_{\mathcal{M}}}{T_{\mathcal{M}\wedge \mathcal{A}}}={\left|I_{n-m}+{\sigma }_{1n}^2{X}_{\mathcal{M}}^{*}{X}_{\mathcal{M}}^{*\top }\right|}^{-\frac{1}{2}}{\left|I_{n-m}+{\sigma }_{1n}^2{X}_{\mathcal{M}\wedge \mathcal{A}}^{*}{X}_{\mathcal{M}\wedge \mathcal{A}}^{*\top }\right|}^{\frac{1}{2}}⪯{\left(n{\sigma }_{1n}^2{\lambda }_1^{*}\right)}^{-\mid \mathcal{M}\mid /2}{\left(n{\sigma }_{1n}^2{\lambda }_2^{*}\right)}^{\mid \mathcal{M}\wedge \mathcal{A}\mid /2}$$

$$\frac{T_{\mathcal{M}\wedge \mathcal{A}}}{T_{\mathcal{A}}}={\left|I_{n-m}+{\sigma }_{1n}^2{X}_{\mathcal{M}\wedge \mathcal{A}}^{*}{X}_{\mathcal{M}\wedge \mathcal{A}}^{*\top }\right|}^{-\frac{1}{2}}{\left|I_{n-m}+{\sigma }_{1n}^2{X}_{\mathcal{A}}^{*}{X}_{\mathcal{A}}^{*\top }\right|}^{\frac{1}{2}}\le {\left|I_{n-m}+{\sigma }_{1n}^2{X}_{\mathcal{A}\wedge {\mathcal{M}}^{*}}^{*}{X}_{\mathcal{A}\wedge \mathcal{M}}^{*\top }\right|}^{\frac{1}{2}}⪯{\left(n{\sigma }_{1n}^2{\lambda }_2^{*}\right)}^{\mid \mathcal{A}\wedge {\mathcal{M}}^c\mid /2}.$$

Thus, $\frac{T_{\mathcal{M}}}{T_{\mathcal{A}}}⪯{\left(n{\sigma }_{1n}^2{\lambda }_1\right)}^{-(\left|\mathcal{M}\right|-\left|\mathcal{A}\right|)/2}{\left(\frac{{\lambda }_2}{{\lambda }_1}\right)}^{\left|\mathcal{A}\right|/2}$, so

$$\begin{gathered} PR(\mathcal{M},\mathcal{A})=\frac{\text{Pr}(\gamma =\mathcal{M}\mid Y,\sigma )}{\text{Pr}(\gamma =\mathcal{A}\mid Y,\sigma )}=q_n^{\left|\mathcal{M}\right|-\left|\mathcal{A}\right|}\frac{T_{\mathcal{M}}}{T_{\mathcal{A}}}\text{exp}\left\{-\frac{1}{2{\sigma }^2}\left(R_{\mathcal{M}}-R_{\mathcal{A}}\right)\right\} \\ ⪯q_n^{\left|\mathcal{M}\right|-\left|\mathcal{A}\right|}{\left(n{\sigma }_{1n}^2{\lambda }_1\right)}^{-(\left|\mathcal{M}\right|-\left|\mathcal{A}\right|)/2}{\left(\frac{{\lambda }_2}{{\lambda }_1}\right)}^{\left|\mathcal{A}\right|/2}\text{exp}\left\{-\frac{1}{2{\sigma }^2}\left(R_{\mathcal{M}}-R_{\mathcal{A}}\right)\right\} \\ ⪯p_n^{-1.5\delta (\left|\mathcal{M}\right|-\left|\mathcal{A}\right|)+0.5\kappa }\text{exp}\left\{-\frac{1}{2{\sigma }^2}\left(R_{\mathcal{M}}-R_{\mathcal{A}}\right)\right\}. \end{gathered}$$

□

Proof of Lemma 2. By Condition A, if Σ_0n is taken to be Σ_0n = {(I_n − CD^⊤D)^{− 1} – I_n}⁺, since Σ_1n is defined as ${\Sigma }_{1n}=I_n-{\left({\Sigma }_{0n}^{+}+I_n\right)}^{-1}$, so we have

$${\Sigma }_{1n}=C{D}^{\top }D,$$

where $D\in {ℝ}^{(n-m)\times n}$ is the difference matrix defined in (2) and 0 < C ≤ min{1, 1/λ_max(D^⊤D)} is a constant, thus Σ_αn = I_n − Σ_1n and ${\Sigma }_{0n}={\left({\Sigma }_{\alpha n}^{-1}-I_n\right)}^{+}$ are semi-positive definite. By taking difference operation on each side, we have

$$Y^{*}=X^{*}+\delta +\mathit{\omega },$$

where $Y^{*}=C^{1/2}DY\in {ℝ}^{n-m}$, $X^{*}=C^{1/2}DX\in {ℝ}^{(n-m)\times p_n}$, $\delta =C^{1/2}D\mathit{\alpha }\in {ℝ}^{n-m}$, $\omega =C^{1/2}D\mathit{ϵ}\in {ℝ}^{n-m}$. The projection matrix is defined as $P_{\mathcal{M}}=X_{\mathcal{M}}^{*}{\left(X_{\mathcal{M}}^{*}{X}_{\mathcal{M}}^{*}\right)}^{-1}{X}_{\mathcal{M}}^{*\top }$ and furthermore we denote $Q_{\mathcal{M}}=I_{n-m}-P_{\mathcal{M}}$. Under the true model, we have $Y=\alpha +X_{\mathcal{A}}{\beta }_{\mathcal{A}}+\mathit{ϵ}$, then for any model containing the true model $\mathcal{A}\subseteq \mathcal{M}$,

$$\begin{gathered} R_{\mathcal{M}}^{*}=Y^{\top }\left\{{\Sigma }_{1n}-{\Sigma }_{1n}{X}_{\mathcal{M}}{\left(X_{\mathcal{M}}^{\top }{\Sigma }_{1n}{X}_{\mathcal{M}}\right)}^{-1}{X}_{\mathcal{M}}^{\top }{\Sigma }_{1n}\right\}Y=Y^{*\top }\left(I_{n-m}-P_{\mathcal{M}}\right)Y^{*} \\ ={(\delta +\omega )}^{\top }{Q}_{\mathcal{M}}(\delta +\omega )={\omega }^{\top }{Q}_{\mathcal{M}}\omega +2{\omega }^{\top }{Q}_{\mathcal{M}}\delta +{\delta }^{\top }{Q}_{\mathcal{M}}\delta . \end{gathered}$$

It suffices to show ${\delta }^{\top }{Q}_{\mathcal{M}}\delta \to 0$ a.s., ${\omega }^{\top }{Q}_{\mathcal{M}}\delta \to 0$ a.s. and ${\sigma }^{-2}{\omega }^{\top }{Q}_{\mathcal{M}}\omega ~C{\chi }_{n-m-\left|\mathcal{M}\right|}^2$ a.s..

Step 1: To show that ${\delta }^{\top }{Q}_{\mathcal{M}}\delta \to 0$ a.s.: note that

$$0\le {\delta }^{\top }{Q}_{\mathcal{M}}\delta =\Vert \delta {\Vert }_2^2-{\Vert P_{\mathcal{M}}\delta \Vert }_2^2\le \Vert \delta {\Vert }_2^2,$$

where ${\delta }_i=C^{1/2}{\sum }_{t=1}^{m+1}{d}_{t}f\left(U_{i+m+1-t}\right)$. Thus by $\sum d_i=0$, $\sum d_i^2=1$, we have

$$\begin{gathered} {\delta }_i^2=C{\left\{\sum _{t=1}^{m+1}{d}_{t}f\left(U_{i+m+1-t}\right)\right\}}^2 \\ =C{\left[d_2\left\{f\left(U_{i+m-1}\right)-f\left(U_{i+m}\right)\right\}+d_3\left\{f\left(U_{i+m-2}\right)-f\left(U_{i+m}\right)\right\}+\dots +d_{m+1}\left\{f\left(U_i\right)-f\left(U_{i+m}\right)\right\}\right]}^2. \end{gathered}$$

By Cauchy-Schwartz inequality and Condition F, for any f (·) ∈ Λ^k(M),

$${\delta }_i^2\le C\left(1-d_1^2\right)\left(\sum _{t=1}^m{M}^2{\Vert U_{i+m-t}-U_{i+m}\Vert }^{2(k\wedge 1)}\right)\le O\left(m^{-1}\right)m^{1+2(k\wedge 1)}{U}_{\infty }^{2(k\wedge 1)}=O\left({\left(m{U}_{\infty }\right)}^{2(k\wedge 1)}\right).$$

By Conditions D, E, F, 1 + (c2 − c1)2(k ∧ 1) < 0, it can be shown that $\Vert \delta {\Vert }_2^2=O\left(n{\left(m{U}_{\infty }\right)}^{2(k\wedge 1)}\right)\to 0$. Then ${\delta }^{\top }{Q}_{\mathcal{M}}\delta \to 0$ a.s..

Step 2: To show that ${\omega }^{\top }{Q}_{\mathcal{M}}\delta \to 0$ a.s.: under fixed design, we have

$${\omega }^{\top }{Q}_{\mathcal{M}}\delta \mid \delta ~\mathcal{N}\left(0,C{\sigma }^2{\delta }^{\top }{Q}_{\mathcal{M}}D{D}^{\top }{Q}_{\mathcal{M}}\delta \right).$$

Since DD^⊤ → I_n−m and ${\delta }^{\top }{Q}_{\mathcal{M}}\delta \to 0$ a.s. from Step 1, so ${\mathit{\omega }}^{\top }{Q}_{\mathcal{M}}\delta \to 0$ a.s.

Step 3: To show that ${\sigma }^{-2}{\omega }^{\top }{Q}_{\mathcal{M}}\omega ~C{\chi }_{n-m-\left|\mathcal{M}\right|}^2$ a.s.: by Condition E, ${\omega }^{\top }{Q}_{\mathcal{M}}\omega =C{ϵ}^{\top }\left(D^{\top }{Q}_{\mathcal{M}}D\right)ϵ$. Let $J=\left(0_{(n-m)\times m},I_{n-m}\right)\in {ℝ}^{(n-m)\times n}$, then ω = C^1/2D_∈ = C^1/2J_∈ + C^1/2(D − J)_∈ = ω₁ + ω₂. Since D → J, so ω₂ is negligible as compared to ω₁ as n goes to infinity. Furthermore ${\omega }_1^{\top }{Q}_{\mathcal{M}}{\omega }_1=C{\mathit{ϵ}}^{\top }{J}^{\top }{Q}_{\mathcal{M}}J\mathit{ϵ}~C{\sigma }^2{\chi }_{n-m-\left|\mathcal{M}\right|}^2$, therefore ${\sigma }^{-2}{\omega }^{\top }{Q}_{\mathcal{M}}\omega ~C{\chi }_{n-m-\left|\mathcal{M}\right|}^2$ a.s..

Overall we have $R_{\mathcal{M}}^{*}~C{\sigma }^2{\chi }_{n-m-\left|\mathcal{M}\right|}^2$ a.s.. Similarly, write $R_{\mathcal{A}}^{*}-R_{\mathcal{M}}^{*}={\omega }^{\top }\left(Q_{\mathcal{A}}-Q_{\mathcal{M}}\right)\omega +2{\omega }^{\top }\left(Q_{\mathcal{A}}-Q_{\mathcal{M}}\right)\delta +{\delta }^{\top }(Q_{\mathcal{A}}-Q_{\mathcal{M}})\delta$. It can be proven that the second and third terms are almost surely 0, so $R_{\mathcal{A}}^{*}-R_{\mathcal{M}}^{*}~{\omega }^{\top }\left(Q_{\mathcal{A}}-Q_{\mathcal{M}}\right)\omega ~C{\sigma }^2{\chi }_{\left|\mathcal{M}\right|-\left|\mathcal{A}\right|}^2$. □

Proof of Lemma 3. We first prove part (i). Note that $R_{\mathcal{A}}=Y^{*\top }\left\{I_{n-m}-X_{\mathcal{A}}^{*}{\left(\frac{1}{{\sigma }_{1n}^2}I+X_{\mathcal{A}}^{*\top }{X}_{\mathcal{A}}^{*}\right)}^{-1}{X}_{\mathcal{A}}^{*\top }\right\}{Y}^{*}$, $R_{\mathcal{A}}^{*}=Y^{*\top }\left\{I_{n-m}-X_{\mathcal{A}}^{*}{\left(X_{\mathcal{A}}^{*\top }{X}_{\mathcal{A}}^{*}\right)}^{-1}{X}_{\mathcal{A}}^{*\top }\right\}{Y}^{*}$, thus

$$\begin{gathered} 0\le R_{\mathcal{A}}-R_{\mathcal{A}}^{*}=Y^{*\top }{X}_{\mathcal{A}}^{*}\left\{{\left(X_{\mathcal{A}}^{*\top }{X}_{\mathcal{A}}^{*}\right)}^{-1}-{\left(\frac{1}{{\sigma }_{1n}^2}I+X_{\mathcal{A}}^{*\top }{X}_{\mathcal{A}}^{*}\right)}^{-1}\right\}{X}_{\mathcal{A}}^{*\top }{Y}^{*} \\ =Y^{*\top }{X}_{\mathcal{A}}^{*}{\left(X_{\mathcal{A}}^{*}{X}_{\mathcal{A}}^{*}\right)}^{-1}{\left\{{\sigma }_{1n}^{2}I+{\left(X_{\mathcal{A}}^{*\top }{X}_{\mathcal{A}}^{*}\right)}^{-1}\right\}}^{-1}{\left(X_{\mathcal{A}}^{*\top }{X}_{\mathcal{A}}^{*}\right)}^{-1}{X}_{\mathcal{A}}^{*\top }{Y}^{*}\le {\sigma }_{1n}^{-2}{Y}^{*\top }{X}_{\mathcal{A}}^{*}{\left(X_{\mathcal{A}}^{*\top }{X}_{\mathcal{A}}^{*}\right)}^{-2}{X}_{\mathcal{A}}^{*}{Y}^{*}, \end{gathered}$$

where the first equality is due to the Woodbury matrix identity A⁻¹ − (A + UCV)⁻¹ = A⁻¹U(C⁻¹ + VA⁻¹U)⁻¹ VA⁻¹. Denote $M=n{X}_{\mathcal{A}}^{*}{\left(X_{\mathcal{A}}^{*\top }{X}_{\mathcal{A}}^{*}\right)}^{-2}{X}_{\mathcal{A}}^{*\top }$, which has rank $\left|\mathcal{A}\right|$ and ${\lambda }_{\text{max}}(M)\le 1/{\lambda }_1^{*}~1/{\lambda }_1$. By [13], we can derive the tail bound for the quadratic term:

$$\text{Pr}\left(R_{\mathcal{A}}-R_{\mathcal{A}}^{*}\ge g_n\right)\le \text{Pr}\left(Y^{*\top }M{Y}^{*}\ge n{\sigma }_{1n}^2{g}_n\right)\le \text{exp}\left(-cn{\sigma }_{1n}^2{\lambda }_1{g}_n\right)$$

We next prove part (ii). By Lemma 2, $R_{\mathcal{A}}^{*}/\left(C{\sigma }^2\right)~{\chi }_{n-m-\left|\mathcal{A}\right|}^2$, by the tail bound for χ² distribution in [17], for any positive x, we have

$$\text{Pr}\left\{\left|\frac{R_{\mathcal{A}}^{*}}{C{\sigma }^2}-(n-m-\left|\mathcal{A}\right|)\right|\ge 2(n-m-\left|\mathcal{A}\right|)(\sqrt{x}+2x)\right\}\le 2\text{exp}\left\{-(n-\left|\mathcal{A}\right|)x\right\}.$$

Furthermore, since m = o(n), thus for any fixed ∈ > 0, there exists a constant c > 0, such that

$$\text{Pr}\left(\left|\frac{R_{\mathcal{A}}^{*}}{nC{\sigma }^2}-1\right|>ϵ\right)\le \text{exp}(-cn).$$

□

Proof of Theorem 4. We will use strategy related that of Theorem 4.1 in [21] to establish Theorem 4.

For overfitted models ${\mathcal{P}}_1=\left\{\mathcal{M}:\mathcal{A}\subseteq \mathcal{M},\left|\mathcal{M}\right|\le m_n+\left|\mathcal{A}\right|\right\}$, we first put bound on $R_{\mathcal{M}}-R_{\mathcal{A}}$. By Lemma 2, for $\mathcal{M}\in {\mathcal{P}}_1$, we have $R_{\mathcal{A}}^{*}-R_{\mathcal{M}}^{*}~C{\sigma }^2{\chi }_{\left|\mathcal{M}\right|-\left|\mathcal{A}\right|}^2$. For any x > 0, $\sqrt{2/3}<w<1$, there exists some constant c > 0, such that

$$\begin{gathered} \text{Pr}\left\{R_{\mathcal{A}}^{*}-R_{\mathcal{M}}^{*}>C{\sigma }^2(2+3x)(\left|\mathcal{M}\right|-\left|\mathcal{A}\right|)\text{ln\hspace{0.17em}}{p}_n\right\}=\text{Pr}\left\{{\chi }_{\left|\mathcal{M}\right|-\left|\mathcal{A}\right|}^2>(2+3x)(\left|\mathcal{M}\right|-\left|\mathcal{A}\right|)\text{ln\hspace{0.17em}}{p}_n\right\} \\ \le \text{Pr}\left\{{\chi }_{\left|\mathcal{M}\right|-\left|\mathcal{A}\right|}^2-(\left|\mathcal{M}\right|-\left|\mathcal{A}\right|)>\left(2+3w^{2}x\right)(\left|\mathcal{M}\right|-\left|\mathcal{A}\right|)\text{ln\hspace{0.17em}}{p}_n\right\} \\ \le c\text{exp}\left\{-(1+x)(\left|\mathcal{M}\right|-\left|\mathcal{A}\right|)\text{ln\hspace{0.17em}}{p}_n\right\}\le c{p}_n^{-(1+x)(\left|\mathcal{M}\right|-\left|\mathcal{A}\right|)}. \end{gathered}$$

Define events $A(\mathcal{M})=\left\{R_{\mathcal{A}}-R_{\mathcal{M}}>2C{\sigma }^2(1+2s)(\left|\mathcal{M}\right|-\left|\mathcal{A}\right|)\text{ln\hspace{0.17em}}{p}_n\right\}$, $U(d)={\cup }_{\mathcal{M}\in {\mathcal{P}}_1:\left|\mathcal{M}\right|=d}A(\mathcal{M})$.

Then for any fixed s > 0, there exists some c, c′ > 0, such that

$$\begin{gathered} \text{Pr}\{U(d)\}=\text{Pr}\{\underset{\mathcal{M}\in {\mathcal{P}}_1:\left|\mathcal{M}\right|=d}{\cup }\left\{R_{\mathcal{A}}-R_{\mathcal{M}}>2C{\sigma }^2(1+2s)(|\mathcal{M}|-|\mathcal{A}|)\text{ln\hspace{0.17em}}{p}_n\right\}] \\ =\text{Pr}\left[\underset{\mathcal{M}\in {\mathcal{P}}_1:\left|\mathcal{M}\right|=d}{\cup }\left\{R_{\mathcal{A}}-R_{\mathcal{M}}+R_{\mathcal{M}}^{*}-R_{\mathcal{M}}^{*}>2C{\sigma }^2(1+2s)(|\mathcal{M}|-|\mathcal{A}|)\text{ln\hspace{0.17em}}{p}_n\right\}\right] \\ \le \text{Pr}\left[\underset{\mathcal{M}\in {\mathcal{P}}_1:\left|\mathcal{M}\right|=d}{\cup }\left\{R_{\mathcal{A}}-R_{\mathcal{M}}^{*}>2C{\sigma }^2(1+2s)(|\mathcal{M}|-|\mathcal{A}|)\text{ln\hspace{0.17em}}{p}_n\right\}\right] \\ \le \text{Pr}\left[\underset{\mathcal{M}\in {\mathcal{P}}_1:\left|\mathcal{M}\right|=d}{\cup }\left\{R_{\mathcal{A}}^{*}-R_{\mathcal{M}}^{*}>C{\sigma }^2(2+3s)(d-|\mathcal{A}|)\text{ln\hspace{0.17em}}{p}_n\right\}\right] \\ +\text{Pr}\left[R_{\mathcal{A}}-R_{\mathcal{A}}^{*}>C{\sigma }^{2}s(d-|\mathcal{A}|)\text{ln\hspace{0.17em}}{p}_n\right], \end{gathered}$$

where the first inequality is due to the fact that $R_{\mathcal{M}}-R_{\mathcal{M}}^{*}\ge 0$. By Lemma 3 and Condition A, it follows that

$$\text{Pr}\left\{U(d)\right\}\le c^{\prime }{p}_n^{-(1+s)(d-\left|\mathcal{A}\right|)}{p}_n^{d-\left|\mathcal{A}\right|}+\text{exp}\left\{-cn{\sigma }_{1n}^2{\lambda }_1(d-\left|\mathcal{A}\right|)\text{ln\hspace{0.17em}}{p}_n\right\}\le 2c^{\prime }{p}_n^{-s(d-\left|\mathcal{A}\right|)}.$$

Then,

$$\text{Pr}\left\{\underset{d>\left|\mathcal{A}\right|}{\cup }U(d)\right\}\le \sum _{d>\left|\mathcal{A}\right|}\text{Pr}\left\{U(d)\right\}\le \frac{2c^{\prime }}{p_n^s-1}\to 0.$$

So consider the high probability event $\left\{{\cap }_{d>\left|\mathcal{A}\right|}U{(d)}^c\right\}$, we have

$$\begin{gathered} \sum _{\mathcal{M}\in {\mathcal{P}}_1}PR(\mathcal{M},\mathcal{A})⪯\sum _{\mathcal{M}\in {\mathcal{P}}_1}{q}_n^{\left|\mathcal{M}\right|-\left|\mathcal{A}\right|}{p}_n^{-(1+\delta )(\left|\mathcal{M}\right|-\left|\mathcal{A}\right|)}\text{exp}\left\{-\frac{1}{2{\sigma }^2}\left(R_{\mathcal{M}}-R_{\mathcal{A}}\right)\right\} \\ ⪯\sum _{\mathcal{M}\in {\mathcal{P}}_1}{q}_n^{\left|\mathcal{M}\right|-\left|\mathcal{A}\right|}{p}_n^{-(1+\delta )(\left|\mathcal{M}\right|-\left|\mathcal{A}\right|)}\text{exp}\left\{C(1+2s)(\left|\mathcal{M}\right|-\left|\mathcal{A}\right|)\text{ln\hspace{0.17em}}{p}_n\right\} \\ ⪯\sum _{\mathcal{M}\in {\mathcal{P}}_1}{q}_n^{\left|\mathcal{M}\right|-\left|\mathcal{A}\right|}{p}_n^{-(1+\delta )(\left|\mathcal{M}\right|-\left|\mathcal{A}\right|)}{p}_n^{C(1+2s)(\left|\mathcal{M}\right|-\left|\mathcal{A}\right|)}. \end{gathered}$$

As 0 < C < 1, set 0 < s < δ/2, then there exists c > 0 such that

$$\sum _{\mathcal{M}\in {\mathcal{P}}_1}PR(\mathcal{M},\mathcal{A})⪯p_n^{-c}\sum _{\left|\mathcal{M}\right|-\left|\mathcal{A}\right|=1}^{p_n}\left(\begin{array}{c}\left|\mathcal{M}\right|-\left|\mathcal{A}\right|\\ p_n\end{array}\right)p_n^{-(\left|\mathcal{M}\right|-\left|\mathcal{A}\right|)}⪯p_n^{-c}\to 0.$$

Consider large and missing some active variables models ${\mathcal{P}}_2=\left\{\mathcal{M}:\mathcal{A}⊈\mathcal{M},K\left|\mathcal{A}\right|<\left|\mathcal{M}\right|\le m_n+\left|\mathcal{A}\right|\right\}$. Define events

$$\begin{gathered} B(\mathcal{M})=\left\{R_{\mathcal{A}}-R_{\mathcal{M}}>2C{\sigma }^2(1+2s)(\left|\mathcal{M}\right|-\left|\mathcal{A}\right|)\text{ln\hspace{0.17em}}{p}_n\right\} \\ \subseteq \left\{R_{\mathcal{A}}-R_{\mathcal{M}\vee \mathcal{A}}>2C{\sigma }^2(1+2s)(\left|\mathcal{M}\right|-\left|\mathcal{A}\right|)\text{ln\hspace{0.17em}}{p}_n\right\}, \end{gathered}$$

$$V(d)=\underset{\mathcal{M}\in {\mathcal{P}}_2:\left|\mathcal{M}\right|=d}{\cup }B(\mathcal{M}).$$

Similar to the proof for ${\mathcal{P}}_1$, there exists c′ > 0, such that

$$\begin{gathered} \text{Pr}\left\{V(d)\right\}\le P\left[\underset{\mathcal{M}\in {\mathcal{P}}_2:\left|\mathcal{M}\right|=d}{\cup }\left\{R_{\mathcal{A}}-R_{\mathcal{M}\vee \mathcal{A}}>2C{\sigma }^2(1+2s)(\left|\mathcal{M}\right|-\left|\mathcal{A}\right|)\text{ln\hspace{0.17em}}{p}_n\right\}\right] \\ \le 2c^{\prime }{p}_n^{-(1+s)(d-\left|\mathcal{A}\right|)}{p}_n^d\le 2c^{\prime }{p}_n^{-\left(1+w^{\prime }\right)d}{p}_n^d\le 2c^{\prime }{p}_n^{-w^{\prime }d}. \end{gathered}$$

Take s = δ/4, we can find such w′ > 0 as long as $\frac{K-1}{K}(1+s)>1$. That is, K > 1 + 4/δ, which is stated in Condition B. It follows that

$$\text{Pr}\left\{\underset{d>K\left|\mathcal{A}\right|}{\cup }V(d)\right\}\le \sum _{d>K\left|\mathcal{A}\right|}\text{Pr}\left\{V(d)\right\}\le 2c^{\prime }{p}_n^{-w^{\prime }K\left|\mathcal{A}\right|}\to 0.$$

Then consider the high probability event $\left\{{\cap }_{d>K\left|\mathcal{A}\right|}V{(d)}^c\right\}$,

$$\begin{gathered} \sum _{\mathcal{M}\in {\mathcal{P}}_2}PR(\mathcal{M},\mathcal{A})⪯\sum _{\mathcal{M}\in {\mathcal{P}}_2}{q}_n^{\left|\mathcal{M}\right|-\left|\mathcal{A}\right|}{p}_n^{-(1+\delta )(\left|\mathcal{M}\right|-\left|\mathcal{A}\right|)}\text{exp}\left\{C(1+2s)(\left|\mathcal{M}\right|-\left|\mathcal{A}\right|)\text{ln\hspace{0.17em}}{p}_n\right\} \\ ⪯\sum _{\mathcal{M}\in {\mathcal{P}}_2}{q}_n^{\left|\mathcal{M}\right|-\left|\mathcal{A}\right|}{p}_n^{-\delta (\mathcal{M}-1)/2}⪯p_n^{-\delta (K-1)/2}\sum _{\left|\mathcal{M}\right|-\left|\mathcal{A}\right|=1}^{p_n}\left(\begin{array}{c}\left|\mathcal{M}\right|-\left|\mathcal{A}\right|\\ p_n\end{array}\right)p_n^{-(\left|\mathcal{M}\right|-\left|\mathcal{A}\right|)}⪯p_n^{-\delta (K-1)/2}\to 0. \end{gathered}$$

For any model $\mathcal{M}$ belonging to the group of underfitted models ${\mathcal{P}}_3=\left\{\mathcal{M}:\mathcal{A}⊈\mathcal{M},\left|\mathcal{M}\right|\le K\left|\mathcal{A}\right|\right\}$, it follows that

$$\begin{gathered} R_{\mathcal{M}}^{*}-R_{\mathcal{M}\vee \mathcal{A}}^{*}={\Vert \left(P_{\mathcal{M}}-P_{\mathcal{M}\vee \mathcal{A}}\right)Y^{*}\Vert }_2^2={\Vert \left(P_{\mathcal{M}}-P_{\mathcal{M}\vee \mathcal{A}}\right)\left(X_{\mathcal{A}}^{*}{\beta }_{\mathcal{A}}+\delta +\omega \right)\Vert }_2^2 \\ \ge {\left\{{\Vert \left(P_{\mathcal{M}}-P_{\mathcal{M}\vee \mathcal{A}}\right)X_{\mathcal{A}}^{*}{\beta }_{\mathcal{A}}\Vert }_2-{\Vert \left(P_{\mathcal{M}}-P_{\mathcal{M}\vee \mathcal{A}}\right)(\delta +\omega )\Vert }_2\right\}}^2. \end{gathered}$$

By Condition B,

$${\Vert \left(P_{\mathcal{M}}-P_{\mathcal{M}\vee \mathcal{A}}\right)X_{\mathcal{A}}^{*}{\beta }_{\mathcal{A}}\Vert }_2={\Vert \left(I-P_{\mathcal{M}}\right)X_{\mathcal{A}}^{*}{\beta }_{\mathcal{A}}\Vert }_2\ge \sqrt{{\Delta }_n(K)}.$$

And on the other hand, ${\Vert \left(P_{\mathcal{M}}-P_{\mathcal{M}\vee \mathcal{A}}\right)(\delta +\omega )\Vert }_2^2={\Vert \left(P_{\mathcal{M}}-P_{{\mathcal{M}}_{\vee }\mathcal{A}}\right)\delta \Vert }_2^2+{\Vert \left(P_{\mathcal{M}}-P_{\mathcal{M}\vee \mathcal{A}}\right)\omega \Vert }_2^2-2{\omega }^{\top }\left(P_{\mathcal{M}}-P_{\mathcal{M}\vee \mathcal{A}}\right)\delta$. Among them, $0\le {\Vert \left(P_{\mathcal{M}}-P_{\mathcal{M}\vee \mathcal{A}}\right)\delta \Vert }_2^2\le 2\left({\Vert P_{\mathcal{M}}\delta \Vert }_2^2+{\Vert P_{\mathcal{M}\vee \mathcal{A}}\delta \Vert }_2^2\right)\le 4\Vert \delta {\Vert }_2^2\to o(1)$. By the similar trick in Lemma 2, it can be shown ${\omega }^{\top }\left(P_{\mathcal{M}}-P_{\mathcal{M}\vee \mathcal{A}}\right)\delta =0$ a.s..

For any 0 < w < 1,

$$\begin{gathered} \text{Pr}\left[\underset{\mathcal{M}\in {\mathcal{P}}_3}{\cup }\left\{R_{\mathcal{M}}^{*}-R_{\mathcal{M}\vee \mathcal{A}}^{*}<{(1-w)}^2{\Delta }_n(K)\right\}\right]\le \text{Pr}\left[\underset{\mathcal{M}\in {\mathcal{P}}_3}{\cup }\left\{{\Vert \left(P_{\mathcal{M}}-P_{\mathcal{M}\vee \mathcal{A}}\right)\omega \Vert }_2>w/2\sqrt{{\Delta }_n(K)}\right\}\right] \\ \le \text{Pr}\left\{{\Vert P_{\mathcal{A}}\omega \Vert }_2>w/2\sqrt{{\Delta }_n(K)}\right\}=\text{Pr}\left\{C{\sigma }^2{\chi }_{\left|\mathcal{A}\right|}^2>w^2/4{\Delta }_n(K)\right\}\le \text{exp}\left\{-c{\Delta }_n(K)/\left|\mathcal{A}\right|\right\}. \end{gathered}$$

The last step follows by the bound for tail with quadratic form. For any 0 < w′ < 1,

$$\begin{gathered} \text{Pr}\left[\underset{\mathcal{M}\in {\mathcal{P}}_3}{\cup }\left\{R_{\mathcal{M}}-R_{\mathcal{M}\vee \mathcal{A}}<\left(1-w^{\prime }\right){\Delta }_n(K)\right\}\right] \\ =\text{Pr}\left[\underset{\mathcal{M}\in {\mathcal{P}}_3}{\cup }\left\{\left(R_{\mathcal{M}}^{*}-R_{\mathcal{M}\vee \mathcal{A}}^{*}\right)+\left(R_{\mathcal{M}\vee \mathcal{A}}^{*}-R_{\mathcal{M}\vee \mathcal{A}}\right)+\left(R_{\mathcal{M}}-R_{\mathcal{M}}^{*}\right)<\left(1-w^{\prime }\right){\Delta }_n(K)\right\}\right] \\ \le \text{Pr}\left[\underset{\mathcal{M}\in {\mathcal{P}}_3}{\cup }\left\{R_{\mathcal{M}}^{*}-R_{\mathcal{M}\vee \mathcal{A}}^{*}\right\}<\left(1-w^{\prime }/2\right){\Delta }_n(K)\right]+\text{Pr}\left[\underset{\mathcal{M}\in {\mathcal{P}}_3}{\cup }\left\{R_{\mathcal{M}\vee \mathcal{A}}^{*}-R_{\mathcal{M}\vee \mathcal{A}}\right\}<-w^{\prime }/2{\Delta }_n(K)\right] \\ \le 2\text{\hspace{0.17em}exp}\left\{-c^{\prime }{\Delta }_n(K)/\left|\mathcal{A}\right|\right\}, \end{gathered}$$

where the first inequality is due to the fact that $R_{\mathcal{M}}-R_{\mathcal{M}}^{*}\ge 0$, and the last inequality follows by the exponential tails of $n{\sigma }_{1n}^2{\lambda }_1\left(R_{\mathcal{M}\vee \mathcal{A}}-R_{\mathcal{M}\vee \mathcal{A}}^{*}\right)$. The proof is similar to Lemma 2 (1).

Let c = 2w, it follows that

$$\begin{gathered} \text{Pr}\left[\underset{\mathcal{M}\in {\mathcal{P}}_3}{\cup }\left\{R_{\mathcal{M}}-R_{\mathcal{A}}<(1-c){\Delta }_n(K)\right\}\right]\le \text{Pr}\left[\underset{\mathcal{M}\in {\mathcal{P}}_3}{\cup }\left\{R_{\mathcal{M}}-R_{\mathcal{M}\vee \mathcal{A}}<(1-w){\Delta }_n(K)\right\}\right]+\text{Pr}\left[\underset{\mathcal{M}\in {\mathcal{P}}_3}{\cup }\left\{R_{\mathcal{A}}-R_{\mathcal{M}\vee \mathcal{A}}>w{\Delta }_n(K)\right\}\right] \\ \le 2\text{\hspace{0.17em}exp}\left\{-c^{\prime }{\Delta }_n(K)/\left|\mathcal{A}\right|\right\}+\text{Pr}\left[\underset{\mathcal{M}\in {\mathcal{P}}_3}{\cup }\left\{R_{\mathcal{A}}^{*}-R_{\mathcal{M}\vee \mathcal{A}}^{*}>w{\Delta }_n(K)\right\}\right]+\text{Pr}\left\{R_{\mathcal{A}}-R_{\mathcal{A}}^{*}>w{\Delta }_n(K)\right\} \\ \le 3\text{\hspace{0.17em}exp}\left\{-c^{\prime }{\Delta }_n(K)/\left|\mathcal{A}\right|\right\}+\text{Pr}\left\{C{\sigma }^2{\chi }_{K\left|\mathcal{A}\right|}^2>w{\Delta }_n(K)\right\}\le 4\text{exp}\left\{-c^{\prime }{\Delta }_n(K)/\left|\mathcal{A}\right|\right\}\to 0, \end{gathered}$$

where the second inequality holds because $R_{\mathcal{A}}-R_{\mathcal{M}\vee \mathcal{A}}=R_{\mathcal{A}}^{*}-R_{\mathcal{M}\vee \mathcal{A}}^{*}+R_{\mathcal{A}}-R_{\mathcal{A}}^{*}+R_{\mathcal{M}\vee \mathcal{A}}^{*}-R_{\mathcal{M}\vee \mathcal{A}}$, and the last step follows by Condition B. Then consider the high probability event $\left\{{\cap }_{\mathcal{M}\in {\mathcal{P}}_3}\left(R_{\mathcal{M}}-R_{\mathcal{A}}\right)>(1-c){\Delta }_n(K)\right\}$,

$$\begin{gathered} \sum _{\mathcal{M}\in {\mathcal{P}}_3}PR(\mathcal{M},\mathcal{A})⪯\sum _{\mathcal{M}\in {\mathcal{P}}_3}{q}_n^{\left|\mathcal{M}\right|-\left|\mathcal{A}\right|}{\left(n{\sigma }_{1n}^2{\lambda }_1\right)}^{\mid \mathcal{A}\mid /2}{\left({\lambda }_2/{\lambda }_1\right)}^{\left|\mathcal{A}\right|/2}\text{exp}\left\{-(1-c){\Delta }_n(K)/\left(2{\sigma }^2\right)\right\} \\ ⪯\text{exp}\left[-\frac{1}{2{\sigma }^2}\left\{(1-c){\Delta }_n(K)-{\sigma }^2\left|\mathcal{A}\right|(2+3\delta )\text{ln\hspace{0.17em}}{p}_n-{\sigma }^2\left|\mathcal{A}\right|(2+\kappa )\text{ln\hspace{0.17em}}{p}_n\right\}\right]. \end{gathered}$$

By Condition B, ${\Delta }_n(K)>{\sigma }^2\left|\mathcal{A}\right|\text{ln\hspace{0.17em}}{p}_n(4+4\delta +\kappa )$, so, we can find 0 < c < 1, w′ > 0, such that $(1-c){\Delta }_n(K)-{\sigma }^2\left|\mathcal{A}\right|\text{ln\hspace{0.17em}}{p}_n(4+3\delta +\kappa )=w^{\prime }{\sigma }^2\left|\mathcal{A}\right|\text{ln\hspace{0.17em}}{p}_n$. Thus,

$$\sum _{\mathcal{M}\in {\mathcal{P}}_3}PR(\mathcal{M},\mathcal{A})\le \text{exp}\left(-\frac{1}{2{\sigma }^2}{w}^{\prime }{\sigma }^2\left|\mathcal{A}\right|\text{ln\hspace{0.17em}}{p}_n\right)\to 0.$$

□

Proof of Theorem 5. Similar to Theorem 4.2 in [21], we start from (5) and integrate out σ², thus the posterior probability under unknown σ² is

$$\begin{gathered} \text{Pr}(\mathit{\gamma }=\mathcal{M}\mid Y)\propto \int q_n^{\left|\mathcal{M}\right|}{\sigma }^{-n}{\sigma }_{1n}^{-\left|\mathcal{M}\right|}{\left|X_{\mathcal{M}}^{\top }{X}_{\mathcal{M}}+I_n/{\sigma }_{1n}^2\right|}^{-1/2}\text{exp}\left(-\frac{1}{2{\sigma }^2}{R}_{\mathcal{M}}\right)\pi \left({\sigma }^2\right)d{\sigma }^2 \\ \propto q_n^{\left|\mathcal{M}\right|}{\sigma }_{1n}^{-\left|\mathcal{M}\right|}{\left|X_{\mathcal{M}}^{\top }{X}_{\mathcal{M}}+I_n/{\sigma }_{1n}^2\right|}^{-1/2}{\left(2b_0+R_{\mathcal{M}}\right)}^{-a_0-n/2}. \end{gathered}$$

Remark 8. Inside the integration, ${\sigma }^{-n}\text{exp}\left(-\frac{1}{2{\sigma }^2}{R}_{\mathcal{M}}\right)$ is the dominant term as the sum of squared residuals $R_{\mathcal{M}}$ has an order of O_p(n). The theorem applies to a wider family of prior as long as π(σ²) is O_p(1) and the support is not too strange. This includes some commonly used priors like improper non-informative prior π(σ²) ∝ σ⁻² and the class of folded-noncentral-t prior with fixed hyper-parameters $\pi (\sigma )\propto {\left(1+\frac{{\sigma }^2}{v^2{a}^2}\right)}^{-(v+1)/2}$.

By Lemma 1, we have

$$\frac{\text{Pr}(\gamma =\mathcal{M}\mid Y)}{\text{Pr}(\gamma =\mathcal{A}\mid Y)}\propto p_n^{-1.5\delta (\left|\mathcal{M}\right|-\left|\mathcal{A}\right|)+0.5\kappa }{\left(\frac{2b_0+R_{\mathcal{M}}}{2b_0+R_{\mathcal{A}}}\right)}^{-a_0-\frac{n}{2}}.$$

Define ${\rho }_n=\frac{R_{\mathcal{A}}+2b_0}{nC{\sigma }^2}-1$, then ${\rho }_n=o_p(1)$, since

$$\frac{R_{\mathcal{A}}^{*}+2b_0}{nC{\sigma }^2}-1<{\rho }_n=\frac{R_{\mathcal{A}}+2b_0}{nC{\sigma }^2}-1=\frac{R_{\mathcal{A}}^{*}+2b_0}{nC{\sigma }^2}-1+\frac{R_{\mathcal{A}}-R_{\mathcal{A}}^{*}}{nC{\sigma }^2},$$

$$\text{Pr}\left(\left|{\rho }_n\right|>2ϵ\right)\le \text{Pr}\left(\left|\frac{R_{\mathcal{A}}^{*}}{nC{\sigma }^2}-1\right|>ϵ\right)+\text{Pr}\left(\left|\frac{R_{\mathcal{A}}-R_{\mathcal{A}}^{*}}{nC{\sigma }^2}\right|>ϵ\right)\le 2\text{exp}(-cn).$$

First consider overfitted models $\mathcal{M}\in {\mathcal{P}}_1$. Define $x_n=(\left|\mathcal{M}\right|-\left|\mathcal{A}\right|)\text{ln\hspace{0.17em}}{p}_n/n$, t_n = − ln {1 − 2(1 + 4s)x_n}. Then by Condition D, $x_n\le \frac{\delta }{(4+\delta )(2+\delta )}$. Since $-\text{ln}(1-x)\le \frac{x}{1-x}$, then for any s < δ/16, we have

$$t_n=-\text{ln}\left\{1-2(1+4s)x_n\right\}<\frac{2(1+4s)x_n}{1-2(1+4s)x_n}<2(1+\delta /2)x_n.$$

Similar to the proof in Theorem 4 (1), consider the high probability event $\left\{{\cap }_{d>\left|\mathcal{A}\right|}U{(d)}^c\right\}\cap \left\{\left|{\rho }_n\right|<{ϵ}^{*}\right\}$, where (1 +4s)(1 − ∈*) > (1 + 2s), then

$$\begin{gathered} {\left(\frac{2b_0+R_{\mathcal{M}}}{2b_0+R_{\mathcal{A}}}\right)}^{-a_0-\frac{n}{2}}={\left\{1+\frac{R_{\mathcal{M}}-R_{\mathcal{A}}}{nC{\sigma }^2\left(1+{\rho }_n\right)}\right\}}^{-a_0-\frac{n}{2}}⪯\text{exp}\left\{\left(a_0+\frac{n}{2}\right)t_n\right\} \\ ⪯\text{exp}\left\{\left(a_0+\frac{n}{2}\right)2(1+\delta /2)(\left|\mathcal{M}\right|-\left|\mathcal{A}\right|)\text{ln\hspace{0.17em}}{p}_n/n\right\} \\ ⪯\text{exp}\left\{(1+\delta /2)(\left|\mathcal{M}\right|-\left|\mathcal{A}\right|)\text{ln\hspace{0.17em}}{p}_n\right\}⪯p_n^{(1+\delta /2)(\left|\mathcal{M}\right|-\left|\mathcal{A}\right|)}. \end{gathered}$$

The problem reduces to the same problem in Theorem 4 (1). And for $\mathcal{M}\in {\mathcal{P}}_2$, we can use the same trick, thus we have

$$\sum _{\mathcal{M}\in {\mathcal{P}}_1\cup {\mathcal{P}}_2}\frac{\text{Pr}(\gamma =\mathcal{M}\mid Y)}{\text{Pr}(\gamma =\mathcal{A}\mid Y)}\stackrel{P}{\to }0.$$

For underfitted models $\mathcal{M}\in {\mathcal{P}}_3$, similar to the proof in Theorem 4 (3), consider the high probability event $\left\{{\cap }_{\mathcal{M}\in \mathcal{P}}\left[R_{\mathcal{M}}-R_{\mathcal{A}}>(1-c){\Delta }_n(K)\right]\right\}\cap \left\{\left|{\rho }_n\right|<{ϵ}^{*}\right\}$.

If Δ_n(K) = o(n), by lim_n→∞(1 + 1/n)ⁿ = e,

$$\begin{gathered} {\left(\frac{2b_0+R_{\mathcal{M}}}{2b_0+R_{\mathcal{A}}}\right)}^{-a_0-\frac{n}{2}}={\left\{1+\frac{R_{\mathcal{M}}-R_{\mathcal{A}}}{nC{\sigma }^2\left(1+{\rho }_n\right)}\right\}}^{-a_0-\frac{n}{2}}⪯{\left\{1+\frac{(1-c){\Delta }_n(K)}{nC{\sigma }^2\left(1+{\rho }_n\right)}\right\}}^{-a_0-\frac{n}{2}} \\ ⪯\text{exp}\left\{-\left(\frac{n}{2}+a_0\right)\frac{(1-c){\Delta }_n(K)}{nC{\sigma }^2\left(1+{ϵ}^{*}\right)}\right\}⪯\text{exp}\left\{-\frac{(1-c){\Delta }_n(K)}{2{\sigma }^2\left(1+{ϵ}^{*}\right)}\right\}. \end{gathered}$$

It reduces to the same problem in Theorem 4 (3).

While if Δ_n(K) ⪰ n, it follows that

$${\left(\frac{2b_0+R_{\mathcal{M}}}{2b_0+R_{\mathcal{A}}}\right)}^{-a_0-\frac{n}{2}}⪯{\left\{1+\frac{(1-c){\Delta }_n(K)}{nC{\sigma }^2\left(1+{\rho }_n\right)}\right\}}^{-a_0-\frac{n}{2}}⪯\text{exp}\left\{-c^{\prime }n\text{ln}\frac{{\Delta }_n(K)}{n}\right\}⪯\text{exp}\left(-c^{\prime }n\right),$$

which converges even faster to 0 as n → ∞. □

5. Discussion

Inspired by the difference-based method, we have proposed a new Bayesian approach to select variables in the linear component of the partially linear model. We modify the Bayesian shrinking and diffusing priors (BASAD) [21], and propose the new Bayesian subset modeling with diffusing prior (BSM-DP). The idea is extended from linear models to the partially linear model with the help of the difference-based method. Model selection consistency is proved under the setting with ultra-high dimensional covariates. Compared to BASAD, BSM-DP performances better in identifying the low signal covariates and in shorter computation time, as shown in the supplementary material. Results in the simulation studies show that our method has higher tolerance on the correlation among predictors and requires mild conditions on covariates, compared with other existing methods for variable selection on PLM. The proposed model is less likely to overfit the model, which is also illustrated by the real data example about supermarket. However, like all other Bayesian methods, it has some price to pay. We do need specific assumptions on the error distribution. The computation is relative intense as compared to frequentist penalized methods. Finally, similar to frequentist methods, although we showed the required rates for the hyperparameters of the priors, the practical choices of them in finite sample applications still need fine tuning.

Table 2.

Summarized simulation results for Case 2: p = 1000, n = 200 and X is sampled from normal distribution with compound symmetric correlation matrix. The reported values are means of different performance measures averaged over 500 replications. The methods compared include LASSO and SCAD on partial residuals tuned by BIC (LASSO.BIC, SCAD.BIC), PC-simple algorithm on the partial residuals (PC-PR), threshold partial correlation on partial residuals (TPC-PR, TPC-PR.EBIC), proposed method with model selected by MAP and MPM (BSM-DP.MAP, BSM-DP.MPM). The details of different methods and measures are provided in Section 3.1.1. Results under high correlation ρ = 0.8 are highlighted.

	Method	$p_{{\mathcal{A}}^c}^{\text{max}}$	$p_{\mathcal{A}}^{\text{min}}$	$p_{\mathcal{A}=\mathcal{M}}$	$p_{\mathcal{A}\in \mathcal{M}}$	p1	p4	TP	FP	ME
	LASSO.BIC			0.220	1.000	1.000	1.000	4.000	1.652	0.509
	SCAD.BIC			0.938	1.000	1.000	1.000	4.000	0.070	0.028
f(u) = u2	PC-PR			0.420	0.998	0.998	1.000	3.998	0.766	0.063
ρ = 0.2	TPC-PR			0.406	0.998	0.998	1.000	3.998	0.782	0.063
	TPC-PR.EBIC			0.862	0.998	0.998	1.000	3.998	0.140	0.040
	BSM-DP.MAP (new)	0.077	1.000	0.970	1.000	1.000	1.000	4.000	0.030	0.024
	BSM-DP.MPM (new)			0.970	1.000	1.000	1.000	4.000	0.024	0.024
	LASSO.BIC			0.202	1.000	1.000	1.000	4.000	1.746	0.515
	SCAD.BIC			0.950	1.000	1.000	1.000	4.000	0.054	0.027
f(u) = sin(2πu)	PC-PR			0.368	0.984	0.984	1.000	3.984	0.870	0.098
ρ = 0.2	TPC-PR			0.358	0.986	0.986	1.000	3.986	0.884	0.095
	TPC-PR.EBIC			0.826	0.990	0.990	1.000	3.990	0.208	0.060
	BSM-DP.MAP (new)	0.087	1.000	0.970	1.000	1.000	1.000	4.000	0.036	0.032
	BSM-DP.MPM (new)			0.976	1.000	1.000	1.000	4.000	0.026	0.033
	LASSO.BIC			0.000	1.000	1.000	1.000	4.000	16.174	0.390
	SCAD.BIC			0.386	0.386	0.412	1.000	3.386	0.000	0.585
f(u) = u2	PC-PR			***	***	***	***	***	***	***
ρ = 0.8	TPC-PR			***	***	***	***	***	***	***
	TPC-PR.EBIC			***	***	***	***	***	***	***
	BSM-DP.MAP (new)	0.105	1.000	0.942	1.000	1.000	1.000	4.000	0.068	0.034
	BSM-DP.MPM (new)			0.956	1.000	1.000	1.000	4.000	0.048	0.037
	LASSO.BIC			0.000	1.000	1.000	1.000	4.000	16.114	0.401
	SCAD.BIC			0.426	0.426	0.448	1.000	3.426	0.000	0.574
f(u) = sin(2πu)	PC-PR			***	***	***	***	***	***	***
ρ = 0.8	TPC-PR			***	***	***	***	***	***	***
	TPC-PR.EBIC			***	***	***	***	***	***	***
	BSM-DP.MAP (new)	0.093	1.000	0.970	1.000	1.000	1.000	4.000	0.036	0.032
	BSM-DP.MPM (new)			0.976	1.000	1.000	1.087	4.000	0.026	0.033

^***stands for the cases when a single replication takes more than 48 hours. So 500 replications can not be done in timely manner.