Learning Sparse Deep Neural Networks with a Spike-and-Slab Prior

Abstract

Deep learning has achieved great successes in many machine learning tasks. However, the deep neural networks (DNNs) are often severely over-parameterized, making them computationally expensive, memory intensive, less interpretable and mis-calibrated. We study sparse DNNs under the Bayesian framework: we establish posterior consistency and structure selection consistency for Bayesian DNNs with a spike-and-slab prior, and illustrate their performance using examples on high-dimensional nonlinear variable selection, large network compression and model calibration. Our numerical results indicate that sparsity is essential for improving the prediction accuracy and calibration of the DNN.

1. INTRODUCTION

During the past decade, deep neural networks (DNNs) have achieved great successes in solving many complex machine learning tasks such as pattern recognition and natural language processing. A key factor to these successes is their universal approximation power, while heavily relying on a large number of parameters. The DNNs used in these tasks may consist of hundreds of layers and billions of parameters, see e.g. [1] on image classification. Many DNNs are severely over-parameterized; for some networks only 5% of parameters are enough to achieve acceptable models [2]. Moreover, these over-parameterized networks are no longer well calibrated [3].

To reduce the complexity of the DNN and improve its calibration, sparse deep learning has been considered by some researchers, see e.g. [4, 5, 6, 7]. The approximation power of the sparse DNN has been studied from both frequentist and Bayesian perspectives. From the frequentist perspective, [4] and [5] quantify the approximation error of the sparse DNN in approximating Hölder smooth functions. From the Bayesian perspective, [6] established posterior consistency for the Bayesian DNN with a spike-and slab prior, and [7] established posterior consistency for the Bayesian DNN with a mixture Gaussian prior. Quite recently, the theory for sparse deep learning has also been developed under the framework of variational inference, see e.g. [8, 9, 10]. However, due to the approximation nature of the variational posterior distribution, the resulting statistical inference is often sub-optimal.

This paper studies theoretical properties of the Bayesian sparse DNN with a spike-and-slab prior. It is closely related to [6], which both employ a spike-and-slab prior, but our theory is more general. In particular, our theory is developed under the assumption that the input dimension and the upper bound of the connection weights can increase with the training sample size; and the activation function is Lipschitz continuous, which has covered many popular activation functions such as ReLU, tanh, and sigmoid. In contrast, [6] assumes that the input dimension is of the order O(1), the connections weights are bounded, and the activation function is ReLU. This paper also provides a scalable algorithm for simulating from the posterior distribution of the Bayesian DNN. The numerical results indicate that sparsity is essential for improving the prediction accuracy and model calibration of the DNN.

The remaining part of this paper is organized as follows. Section 2 presents the theoretical results. Section 3 describes a stochastic gradient Markov chain Monte Carlo (SGMCMC) algorithm for Bayesian DNN simulation. Section 4 presents a network compression example. Section 5 concludes the paper with a brief discussion.

2. THEORETICAL RESULTS

Let $D_n={\left\{{\mathit{x}}^{(i)},y^{(i)}\right\}}_{i=1}^n$ denote a training dataset of n i.i.d observations, where ${\mathit{x}}^{(i)}\in {ℝ}^{p_n}$, $y^{(i)}\in ℝ$, and p_n denotes the dimension of the input variable and is assumed to grow with the training sample size n_. We assume that the data are generated from a generalized linear model with the pdf/pmf given by

$$f\left(y\mid \mu *(\mathit{x})\right)=\exp \left\{A\left(\mu *(\mathit{x})\right)y+B\left(\mu *(\mathit{x})\right)+C(y)\right\},$$

where μ∗(x) denotes an unknown function of x, and A(·), B(·) and C(·) are appropriately defined functions. For example, for logistic regression, we have A(μ*) = μ*, $B\left(\mu *\right)=-\log \left(1+e^{{\mu }^{*}}\right)$ and C(y) = 1; and for normal regression, by introducing an extra dispersion parameter σ², we have A(μ∗) = μ∗/σ², B(μ∗) = μ∗²/2σ² and C(y) = −y²/2σ² − log(2πσ²)/2. For simplicity of analysis, σ² is assumed to be known. To model the data, we propose to approximate μ∗(x) by a DNN with H_n − 1 hidden layers. Let L_h denote the number of hidden units at layer h, with $L_{H_n}=1$ for the output layer and L₀ = p_n for the input layer. Let ${\mathit{w}}^h\in {ℝ}^{L_h\times L_{h-1}}$ and ${\mathit{b}}^h\in {ℝ}^{L_h\times 1}$, h ∈ {1, 2, ..., H_n} denote the weights and bias of layer h, respectively. Let ${\psi }^h:{ℝ}^{L_h\times 1}\to {ℝ}^{L_h\times 1}$ denote a mapping which applies a piece-wise differentiable activation function ψ to each entry of the input. In summary, the DNN forms a nonlinear mapping

$$\mu (\mathit{\beta },\mathit{x})={\mathit{w}}^{H_n}{\psi }^{H_n-1}\left[\cdots {\psi }^1\left[{\mathit{w}}^1\mathit{x}+{\mathit{b}}^1\right]\cdots \right]+{\mathit{b}}^{H_n},$$ (1)

where $\mathit{\beta }=\left\{{\mathit{w}}_{ij}^h,{\mathit{b}}_k^h:1\le h\le H_n,1\le i,k\le L_h,1\le j\le L_{h-1}\right\}$ denotes the collection of weights and biases of the DNN. A fully connected DNN consists of $K_n=|\mathit{\beta }|={\sum }_{h=1}^{H_n}\left(L_{h-1}\times L_h+L_h\right)$ tuning parameters. To facilitate representation for sparse DNNs, we re-parameterize β by introducing an indicator variable for each element of β, which indicates the existence of the corresponding connection. Here the bias is treated as a special connection with a constant input of 1. Let ${\mathit{w}}^h={\tilde{\mathit{w}}}^h\circ {\mathit{\gamma }}^{w^h}$ and ${\mathit{b}}^h={\tilde{\mathit{b}}}^h\circ {\mathit{\gamma }}^{{\mathit{b}}^h}$, where ${\mathit{\gamma }}^{w^h}$ and ${\mathit{\gamma }}^{{\mathit{b}}^h}$ are the matrices or vectors of indicators, and ◦ denotes element-wise product. Correspondingly, we define $\tilde{\mathit{\beta }}=\left\{{\tilde{w}}_{ij}^h,{\tilde{\mathit{b}}}_k^h\right\}$ and $\mathit{\gamma }=\left\{{\mathit{\gamma }}_{ij}^{w^h},{\mathit{\gamma }}_k^{{\mathit{b}}^h}\right\}$ which represent connection weights and network structure, respectively. For notational simplicity, we rewrite $\tilde{\mathit{\beta }}$ and γ as $\tilde{\mathit{\beta }}=\left({\tilde{\mathit{\beta }}}_1,{\tilde{\mathit{\beta }}}_2,\dots ,{\tilde{\mathit{\beta }}}_{K_n}\right)\in {ℝ}^{K_n}$ and $\mathit{\gamma }=\left({\mathit{\gamma }}_1,{\mathit{\gamma }}_2,\dots ,{\mathit{\gamma }}_{K_n}\right)\in {\{0,1\}}^{K_n}$. Correspondingly, we have $\mathit{\beta }=\tilde{\mathit{\beta }}\circ \mathit{\gamma }$.

2.1. Regularity conditions on sparse DNNs

With slight abuse of notation, we will rewrite μ(β, x) in (1) for the sparse DNN as μ(β, γ, x) by including the network structure information. In what follows, we characterize how well μ∗(x) can be approximated by μ(β, γ, x) with a sparse DNN from the Bayesian perspective. For this purpose, we assume the following regularity conditions:

A.1 The input x is bounded by 1 entry-wisely, i.e.$\mathit{x}\in \mathrm{\Omega }={[-1,1]}^{p_n}$, and the density of x is bounded in its support Ω uniformly with respect to n.

A.2 The unknown regression mean function μ∗(x) can be well approximated by a sparse DNN model such that μ(β∗, γ∗, x) satisfies

• A.2.1${\Vert \mu *(\mathit{x})-\mu \left(\mathit{\beta }*,\mathit{\gamma }*,\mathit{x}\right)\Vert }_{L^2(\mathrm{\Omega })}\le {\varpi }_n$, where the approximation error ϖ_n → 0 as the sample size n → ∞.

• A.2.2 H_n log K_n ≥ C log n for some constant C > 0, $1\prec r_n^{*}{H}_n\log K_n\prec n^1$ and ‖β∗‖_∞ < E_n, where E_n is a sequence satisfying log E_n = O(log K_n), and $r_n^{*}$ is the network size, i.e.,$r_n^{*}={\sum }_{i=1}^{K_n}{\mathit{\gamma }}_i^{*}$.

A.3 The activation function ψ is Lipschitz continuous with a Lipschitz constant of 1.

Refer to the supplementary material for more discussions on the conditions, where we show how our Bayesian DNN theory is compatible with the existing sparse DNN approximation theory.

2.2. The Spike-and-Slab Prior

To conduct Bayesian inference for the DNN, we consider a hierarchical prior given by

$${\tilde{\mathit{\beta }}}_i\mid {\mathit{\gamma }}_i=1~N\left(0,{\sigma }_{1,n}^2\right),i=1,2,\dots ,K_n;\pi (\mathit{\gamma })\propto {\lambda }_n^{\left|\mathit{\gamma }\right|}{\left(1-{\lambda }_n\right)}^{K_n-\left|\mathit{\gamma }\right|}\mathbf{1}\left\{\left|\mathit{\gamma }\right|\le {\overline{r}}_n,\mathit{\gamma }\in \mathcal{G}\right\},$$ (2)

where ${\sigma }_{1,n}^2$ is the variance of the normal distribution, ${\overline{r}}_n$ is the maximum size of candidate sparse DNNs, λ_n can be read as an approximate prior probability for each connection to be included in the DNN, and $\mathcal{G}$ is the set of all valid DNNs. With this hierarchical prior, βi follows a discrete spike-and-slab prior, i.e., ${\mathit{\beta }}_i={\tilde{\mathit{\beta }}}_i\circ {\mathit{\gamma }}_i~{\lambda }_{n}N\left(0,{\sigma }_{1,n}^2\right)+\left(1-{\lambda }_n\right){\delta }_0$ for i = 1, 2, . . . , K_n, where δ₀ denotes the Dirac delta function.

2.3. Posterior consistency

Posterior consistency measures the speed of posterior concentration around the true density function, which plays a major role in validating Bayesian methods. For DNNs, since the total number of parameters K_n is often much larger than the sample size n, posterior consistency provides a general guideline for prior setting. Otherwise, the prior information may dominate data information, rendering a biased inference for the underlying true model. For learning sparse DNNs, we generally set λ_n → 0 as K_n → ∞, which provides an automatic control for the multiplicity involved in structural selection [11]. Denote by P∗ and E∗ the respective probability measure and expectation with respective to the data D_n, denote by d(p₁, p₂) the Hellinger distance between two densities p₁(x, y) and p₂(x, y), denote by p_β the density function induced by a DNN associated with the parameters β, and denote by $p_{\mu *}$ the true density function induced by the unknown function μ∗(x). Let π(A|D_n) denote the posterior probability of an event A. The following theorem establishes posterior consistency for sparse DNNs under the spike-and-slab prior (2).

Theorem 2.1.

Consider a DNN with H_n layers and at most K_n possible connections, where both H_n and K_n are increasing with n. If the assumptions A.1-A.3 hold, and $\tilde{\mathit{\beta }}$ and γ are subject to the prior (2) with the hyperparameters subject to the constraints $\log {\sigma }_{1,n}^2+E_n^2/{\sigma }_{1,n}^2=O\left(H_n\log K_n\right)$ and $1\le r_n\le {\overline{r}}_n\prec K_n$, where r_n = λ_nK_n. Then there exists a sequence ${ϵ}_n\asymp \max \left\{{\varpi }_n,\sqrt{{\overline{r}}_n{H}_n\log K_n/n}\right\}$ such that for sufficiently large n ², the posterior distribution satisfies

$$\begin{array}{l}P*\left\{\pi \left[d\left(p_{\mathit{\beta }},p_{{\mu }^{*}}\right)>4{ϵ}_n\mid D_n\right]\ge 2e^{-n{ϵ}_n^2/4}\right\}\le 2e^{-n{ϵ}_n^2/4},\\ E*\pi \left[d\left(p_{\mathit{\beta }},p_{{\mu }^{*}}\right)>4{ϵ}_n\mid D_n\right]\le 4e^{-n{ϵ}_n^2/2},\end{array}$$ (3)

Theorem 2.1 establishes a Bayesian contraction rate for sparse DNNs under the Hellinger metric. Therefore, to obtain consistent Bayesian inference, it suffices to require ϖ_n → 0 and ${\overline{r}}_n{H}_n\log \left(K_n\right)\prec n$. If μ∗ can be optimally represented by a sparse DNN model (refer to Remark S1 in the supplementary material), then ϖ_n polynomially decays to 0 as r_n increases to infinity. On the other hand, ${\overline{r}}_n{H}_n\log \left(K_n\right)\prec n$ implies ${\overline{r}}_n=o(n/\log (n))$ by noting that $K_n\ge {\overline{r}}_n$. Refer to the supplementary material for more discussions on this theorem. In conclusion, we only need ${\overline{r}}_n$, the upper bound of total connectivity, to increase slowly; or said differently, a sparse DNN of connectivity O(n/ log(n)) could have been large enough for providing a consistent approximation to the true posterior distribution.

2.4. Consistency of DNN structure selection

It is known that the DNN model is nonidentifiable in general due to the symmetry of the network structure. However, by introducing appropriate constrains [12, 13], we can define a set of neural networks such that each network in the set is unique up to simple node permutations, sign changes, etc. Let Θ denote such a set of DNNs, where each element in Θ can be viewed as an equivalent class of networks. Let ν(·) be the transformation such that ν(β) ∈ Θ. Since p_β = p_ν(β), the posterior consistency (3) holds for ν(β) as well. In what follows, we rewrite ν(β) by ν(γ, β) by including the network structure information. To serve the purpose of structure selection for the DNN, we consider the marginal inclusion posterior probability (MIPP) approach proposed in [14] for high-dimensional variable selection. For each entry of γ, we define its MIPP by $q_i=\int {\sum }_{\mathit{\gamma }}{e}_{i\mid v(\mathit{\gamma },\mathit{\beta })}\pi \left(\mathit{\gamma }\mid \mathit{\beta },D_n\right)\pi \left(\mathit{\beta }\mid D_n\right)d\mathit{\beta }$, where e_i|ν(γ,β) indicates existence of connection i in the network ν(γ, β). Similarly, we define $e_{i\mid v\left({\mathit{\gamma }}^{*},{\mathit{\beta }}^{*}\right)}$ as the indicator for connection i in the true DNN ν(γ∗, β∗). The MIPP approach is to choose the connections for which the marginal inclusion posterior probability is greater than a threshold value $\widehat{q}$; that is, setting ${\widehat{\mathit{\gamma }}}_{\widehat{q}}=\left\{j:q_j>\widehat{q},j=1,2,\dots ,K_n\right\}$ as a Bayes estimator of the true model. To establish the consistency of ${\widehat{\mathit{\gamma }}}_{\widehat{q}}$, the following identifiability condition for the true network γ∗ is needed. Let $A\left({ϵ}_n\right)=\left\{\mathit{\beta }:d\left(p_{\mathit{\beta }},p_{{\mu }^{*}}\right)\ge {ϵ}_n\right\}$. Define $\rho \left({ϵ}_n\right)={\max }_{1\le i\le K_n}{\int }_{A{\left({ϵ}_n\right)}^c}{\sum }_{\mathit{\gamma }}\left|e_{i\mid v(\mathit{\gamma },\mathit{\beta })}-e_{i\mid v\left({\mathit{\gamma }}^{*},{\mathit{\beta }}^{*}\right)}\right|\pi \left(\mathit{\gamma }\mid \mathit{\beta },D_n\right)\pi \left(\mathit{\beta }\mid D_n\right)d\mathit{\beta }$, which measures the distance between the true DNN model and the models drawn from the posterior in the ϵ_n-neighborhood A(ϵ_n)^c. The identifiability condition can be stated as follows: $\text{B.1}\rho \left({ϵ}_n\right)\to 0,\text{as}n\to \infty \text{and}{ϵ}_n\to 0.$

That is, when n is sufficiently large, if a DNN produces approximately the same probability distribution as the true data, then its structure, after mapping into the network space Θ, must coincide with the true DNN structure. Note that this identifiability is different from the one mentioned at the beginning of this section, which is only with respect to structure and parameter rearrangements. Theorem 2.2 concerns consistency of γ${\widehat{\mathit{\gamma }}}_{\widehat{q}}$ and its sure screening property.

Theorem 2.2.

If the conditions of Theorem 2.1 and condition B.1 hold, then (i) ${\max }_{1\le i\le K_n}\left\{\left|q_i-e_{i\mid v\left({\mathit{\gamma }}^{*},{\mathit{\beta }}^{*}\right)}\right|\right\}\to 0$ in probability; and (ii) for any pre-specified $\widehat{q}\in (0,1)$,${\lim }_{n\to \infty }P\left({\mathit{\gamma }}_{*}\subset {\widehat{\mathit{\gamma }}}_{\widehat{q}}\right)=1$.

In practice, the value of $\widehat{q}$ can be determined by a multiple hypothesis test, see e.g. [15]. For simplicity, we often set $\widehat{q}=0.5$. Recall that ${\mathit{\gamma }}^{{\mathit{w}}^h}\in {ℝ}^{L_h\times L_{h-1}}$ denotes the connection indicator matrix of layer h. Let

$${\mathit{\gamma }}^{\mathit{x}}={\mathit{\gamma }}^{{\mathit{w}}^{H_n}}{\mathit{\gamma }}^{{\mathit{w}}^{H_n-1}}\cdots {\mathit{\gamma }}^{w^1}\in {ℝ}^{1\times p_n},$$ (4)

and let ${\mathit{\gamma }}_i^{\mathit{x}}$ denote the i-th element of γ^x. If ${\mathit{\gamma }}_i^{\mathit{x}}>0$ then the input variable x_i is effective in the network γ, and ${\mathit{\gamma }}_i^{\mathit{x}}=0$ otherwise. As in treating network connections, we can define a MIPP for each input variable and determine a threshold value for the MIPP based on a multiple hypothesis test. As implied by (4), the consistency of structure selection implies consistency of variable selection with respect to the true network γ∗ as defined in Assumption A.2.1.

3. COMPUTATIONAL ALGORITHM

Although the spike-and-slab prior leads to nice theoretical properties for Bayesian sparse DNNs, the resulting posterior is a mixture of distributions with varying dimensions. Conventional Bayesian sampling algorithms adopt reversible jump moves [16] to explore different network architectures, see e.g. [13], and lack scalability necessarily required for big data problems. [17] proposed an extended stochastic gradient Langevin dynamics (SGLD) algorithm for Bayesian variable selection under the big data scenario, which allows the mini-batch data to be used in updating the model and regression coefficients at each iteration and is thus scalable to big data problems. In this paper, we extend it further by combining it with stochastic gradient Hamiltonian Monte Carlo (SGHMC) [18], see Algorithm 1 for the procedure and the supplementary material for more explanations.

The main parameters of the extended SGHMC algorithm are the prior hyperparameters λ_n and σ_1,n defined in (2). Our theory allows σ_1,n to grow with n from the perspective of data fitting, but in our experience, the magnitude of weights tend to adversely affect the generalization ability of the network. For this reason, we usually set σ_1,n to a small number such as 0.01 or 0.02, and then tune the value of λ_n for network sparsity such that the sparse constraint given in Assumption A.2.2 is satisfied.

4. EXPERIMENTS

This section presents an example on network compression for convolutional neural networks (CNNs). Refer to the supplementary material for examples on high-dimensional nonlinear variable selection and DNN model calibration. In terms of network compression, extended SGHMC can be viewed as an unstructured dynamic pruning algorithm, where the spike-and-slab prior is imposed on each parameter and the sparsity mask γ can be adjusted during training [19]. We test the proposed method on the CIFAR-10 data [20], pruning ResNet20 and ResNet32 networks [1] to different sparsity levels. We train the network using the standard setup as in [19], i.e. we use a mini-batch size of 128 and train the model for 300 epochs with a fixed initial learning rate of ϵ₀ = 0.1 and ϵ_t being divided by 10 at epochs 150 and 225. We set ${\sigma }_{1,n}^2=0.04$, and sample from the posterior using the extended SGHMC algorithm with t₀ = 1,

Algorithm 1.

Extended Stochastic Gradient Hamiltonian Monte Carlo

Input: T ; m; τ; t0; t1

Initialization: Randomly initialize${\tilde{\mathit{\beta }}}^{(1)}$. Set ${\mathit{\gamma }}_i^{\left(0,t_0\right)}=1$ for all i ∈ {1, . . . , Kn};

for t = 1, 2, ..., T do

(i) Draw a subsample of size m from the full dataset Dn, and duplicate it to form a dataset$D_{m,n}^{(t)}$.

(ii) Simulate models ${\mathit{\gamma }}^{(t,1)},{\mathit{\gamma }}^{(t,2)},\dots ,{\mathit{\gamma }}^{\left(t,t_0\right)}$ from the conditional posterior $\pi \left(\mathit{\gamma }\mid {\tilde{\mathit{\beta }}}^{(t)},D_{m,n}^{(t)}\right)$ by MetropolisHastings(MH) algorithm with a magnitude-based proposal.

(iii) Update ${\tilde{\mathit{\beta }}}^{(t)}$ via a SGHMC move,

$${\mathit{v}}^{(t+1)}=(1-\alpha ){\mathit{v}}^{(t)}+\frac{{ϵ}_{t+1}}{2t_0}\sum _{k=1}^{t_0}{\nabla }_{\tilde{\mathit{\beta }}}\log \pi \left({\tilde{\mathit{\beta }}}^{(t)}\mid {\mathit{\gamma }}^{(t,k)},D_{m,n}^{(t)}\right)+\sqrt{\mathit{\alpha }{ϵ}_{t+1}}\tau {\eta }_{t+1},$$ $${\tilde{\mathit{\beta }}}^{(t+1)}={\tilde{\mathit{\beta }}}^{(t)}+{\mathit{v}}^{(t+1)},$$

where ηt+1 ~ N(0, Id), ϵt+1 is the learning rate, τ is the temperature, and 1 − α gives the momentum parameter.

end for

τ = 0.0001, α = 0.1, and a magnitude-based proposal for γ. We tune λ_n and B (a parameter of the magnitude-based proposal as described in the supplementary material) to achieve target sparsity levels for different ResNets. Posterior samples are collected at the end of each epoch and the first 225 samples are discarded for the burn-in process.

We compare the proposed method with some recent unstructured compression methods including dynamic pruning with feedback (DPF) [19], dynamic sparse reparameterization (DSR) [21], sparse momentum (SM) [22], consistent sparse deep learning (BNN_cs) [7], and variation inference for sparse deep learning (SVI) [9]. The numerical results are summarized in Table 1. The comparison shows that the proposed method can achieve higher prediction accuracy than the existing ones at a similar sparsity level.

Table 1:

Network pruning for CIFAR-10 data, where the result of each method is calculated by averaging over 3 independent runs with the standard deviation reported in the parentheses. BNN_bma and BNN_single denote the proposed method with the results calculated in the way of Bayesian model averaging and based on the single model collected at the last epoch, respectively. Other methods have been defined in the context.

Model	Method	Pruning Ratio(%)	Test Accuracy(%)	Pruning Ratio(%)	Test Accuracy(%)
ResNet20	BNNbma	9.65(0.05)	91.60(0.06)	19.76(0.02)	92.65(0.02)
	BNNsingle	9.88(0.08)	91.26(0.02)	19.83(0.02)	92.32(0.04)
	BNNcs	9.55(0.03)	91.27(0.05)	19.67(0.05)	92.27(0.03)
	SVI	10	89.43(0.05)	20	91.78(0.06)
	SM	10	89.76(0.40)	20	91.54(0.16)
	DSR	10	87.88(0.04)	20	91.78(0.28)
	DPF	10	90.88(0.07)	20	92.17(0.21)
ResNet32	BNNbma	4.89(0.09)	91.84(0.09)	9.65(0.05)	92.99(0.08)
	BNNsingle	4.99(0.06)	91.39(0.10)	8.77(0.12)	92.74(0.10)
	BNNcs	4.78(0.01)	91.21(0.01)	9.53(0.04)	92.74(0.07)
	SVI	5	86.31(0.23)	10	91.61(0.10)
	SM	5	88.68(0.22)	10	91.54(0.18)
	DSR	5	84.12(0.32)	10	91.41(0.23)
	DPF	5	90.94(0.35)	10	92.42(0.18)

5. CONCLUSION

In this paper, we study sparse deep learning from the Bayesian perspective. Theoretically, we establish posterior consistency and structure selection consistency for Bayesian DNNs with a spike-and-slab prior. We give the posterior contraction rate, and propose to use the marginal inclusion posterior probability approach to determine the structure of the sparse DNN. We employ a scalable SGMCMC algorithm for simulating from the posterior of the sparse DNN, which has about the same order of computational complexity as popular optimization algorithms such as SGD. When irrelevant features exist in the dataset, our numerical results show that learning a sparse DNN is always rewarded in prediction compared to blindly learning a large, fully connected DNN. The proposed method can be applied for large-scale network pruning tasks as an unstructured dynamic pruning method, where the sparsity mask is adjusted systematically during training. Our numerical result show that the proposed method can achieve the state-of-the-art performance in network structure selection, large-scale network pruning and model calibration.

As mentioned previously, sparsity is essential for improving the calibration of the DNN [23, 24]. This work provides a practical and efficient method for learning sparse DNNs with a theoretical guarantee for identification of an appropriate sparse structure, and thus it will naturally enhance the development of trustworthy artificial intelligence.