On exact Bayesian credible sets for discrete parameters

Abstract

We introduce a generalized Bayesian credible set that can achieve any preassigned credible level, addressing a limitation of the current credible sets. This is achieved by exploiting a connection between the highest posterior density set and the Neyman-Pearson lemma.

1. Introduction

The credible set is one of the most useful tools in Bayesian inference to quantify the uncertainty in Bayesian estimation. See, for example, [1], and [2]. It plays a similar role as the confidence set in the frequentist setting. The most commonly used credible set is the highest posterior density credible set, which has the shortest length for a given credible level. Approximate Bayesian credible sets can be constructed using the Bernstein-von Mises theorem: see, for example, [3] and [4].

However, the classical definition of the highest posterior density credible set [1] is useful mostly for continuous parameters—the parameter supported on an interval or a set with a nonempty and connected interior. In the discrete case where the parameter is supported on a finite or a countable set, the highest posterior density credible set may not exist for a prescribed credible level. This problem is particularly acute for Bayesian classification where the highest posterior density credible set almost never exists for a given credible level. To our knowledge, there has been no systematic method to construct the shortest credible set for such problems.

Bayesian classification has been widely used in many contemporary scientific studies such as cancer research, genomics, and healthcare applications. See, for example, [5], [6],[7],[8], [9], and [10]. Thus, the need for precise uncertainty quantification for Bayesian classification is prevalent. It is high time to facilitate Bayesian classification with exact inference tools and bring it up to par with Bayesian inference for continuous parameters, where exact credible sets have long been available and widely used. The current paper is one step towards satisfying this need.

Our extension is based on the following insight: the process of identifying the highest posterior density credible set is rather similar to that of constructing the most powerful test based on the Neyman-Pearson lemma, and the latter can accommodate any prescribed significance level—even for discrete random variables—by strategically placing probability masses at some transitioning points. This consideration leads us to define the generalized highest posterior density credible set not as a set in the parameter space, but as a mapping from the parameter space to the closed interval [0, 1]. By this adjustment, there always exists a unique shortest credible set for any prescribed credible level, regardless of the nature of the parameter. Accompanying this methodology, we also introduce (in the Supplementary Material) a visualization tool for classification called the Steering Wheel Plot.

2. Generalized credible set

Definition 2.1.

For an $\alpha \in (\mathrm{0,1})$, we call any measurable function $\varphi :{\text{Ω}}_{\text{Θ}}\times {\text{Ω}}_X\to [\mathrm{0,1}]$ such that

$$E\{\varphi (\text{Θ}\mid X)\mid X\}=1-\alpha \text{a.s.}{P}_X$$

$\text{a}(1-\alpha )$-level generalized credible set for $\theta$.

This definition is motivated by the randomized test in classical hypothesis testing theory (see, for example, [11]). Here, the value of $\varphi (\text{Θ}\mid X)$ can be interpreted as follows: conditioning on the data $X$, at each value of $\text{Θ}$, we perform a Bernoulli trial with success probability $\varphi (\text{Θ}\mid X)$; if the trial turns up head, we classify $\text{Θ}$ as belonging to the credible set; otherwise, we classify $\text{Θ}$ as not belonging to the credible set. Alternatively, we can also interpret $\varphi (\text{Θ}\mid X)$ as the membership function in fuzzy logic (see, for example, [12], [13], and [14]), where a larger value $\varphi (\text{Θ}\mid X)$ represents a stronger tendency of belonging to the credible set.

Under this new framework, the classical credible set, which is a subset of ${\text{Ω}}_{\text{Θ}}$, can be viewed as a mapping $\varphi$ from ${\text{Ω}}_{\text{Θ}}\times {\text{Ω}}_X$ to the set {0, 1}; that is, $\varphi$ is the indicator function of the credible set. Since $\{0,1\}\subseteq [0,1]$, the generalized credible set indeed generalizes the credible set.

3. Generalized highest posterior density credible set

As mentioned earlier, the highest posterior density credible set is the smallest credible set in the sense that it has the smallest ${\mu }_{\text{Θ}}$-measure among all credible sets of the same level. More precisely, if $C\left({\kappa }_{\alpha }\right)$ is the highest posterior density credible set of level $1-\alpha$ and $C$ is any other credible set of level $1-\alpha$, then ${\mu }_{\text{Θ}}\{C({\kappa }_{\alpha })\}\le {\mu }_{\text{Θ}}(C)$. In this section, we extend this optimal property to generalized credible sets. We first generalize the highest posterior density credible set.

Definition 3.1.

Let $\gamma (\theta \mid x)$ be a value in the range [0, 1]. If there is a constant ${\kappa }_{\alpha }(x)>0$ such that the generalized credible set

$${\varphi }^{*}(\theta \mid x)=\left\{\begin{array}{ll}1& \text{if}{\pi }_{\text{Θ}\mid X}(\theta \mid x)>{\kappa }_{\alpha }(x)\\ \gamma (\theta \mid x)& \text{if}{\pi }_{\text{Θ}\mid X}(\theta \mid x)={\kappa }_{\alpha }(x)\\ 0& \text{if}{\pi }_{\text{Θ}\mid X}(\theta \mid x)<{\kappa }_{\alpha }(x)\end{array}\right.$$ (1)

satisfies $E\{{\varphi }^{*}(\text{Θ}\mid X)\mid X\}=1-\alpha$, then ${\varphi }^{*}$ is called the level $1-\alpha$ generalized highest posterior density credible set.

If we take $\gamma (\theta \mid x)$ to be constantly 1, the above function is nothing but the indicator function of the set $\{\theta :{\pi }_{\text{Θ}\mid X}(\theta \mid x)\ge {\kappa }_{\alpha }\}$, which is the definition of the highest posterior density credible set. The difference is that for the highest posterior density credible set we can only guarantee $P[{\text{Θ}}^{-1}\{C({\kappa }_{\alpha })\}\mid X]\ge 1-\alpha$, where the equality may not be achieved. In comparison, as we will show in Theorem 3.3, the exact level $1-\alpha$ can always be achieved by the generalized highest posterior density credible set.

Now that the measure ${\mu }_{\text{Θ}}(C)$ is replaced by the integral $\int \varphi (\theta \mid x)d{\mu }_{\text{Θ}}(\theta )$, the size of a generalized credible set has a new meaning: we simply use $\int \varphi (\theta \mid x)d{\mu }_{\text{Θ}}(\theta )$ to define the size of a generalized credible set with $\varphi (\theta \mid x)$. The next two theorems show that the generalized highest posterior density credible set is the smallest generalized credible set, that it exists, and that it is unique. The proofs of the theorems echo the proof of the Neyman-Pearson lemma.

Theorem 3.2.

If ${\varphi }^{*}:{\text{Ω}}_{\text{Θ}}\times {\text{Ω}}_X\to [\mathrm{0,\; 1}]$ is the level $1-\alpha$ generalized highest posterior density credible set, then it is the smallest level $1-\alpha$ generalized credible set; that is, for any $\varphi :{\text{Ω}}_{\text{Θ}}\times {\text{Ω}}_X\to [\mathrm{0,\; 1}]$ satisfying $E\{\varphi (\text{Θ}\mid X)\mid X\}=1-\alpha$, we have

$$\int {\varphi }^{*}(\theta \mid x)d{\mu }_{\text{Θ}}(\theta )\le \int \varphi (\theta \mid x)d{\mu }_{\text{Θ}}(\theta ).$$

We next show the existence of the generalized highest posterior density credible set and the uniqueness of the smallest generalized credible set up to the choice of $\gamma (\theta \mid x)$. The specific form of $\gamma (\theta \mid x)$ is established in Theorem 4.2.

Theorem 3.3.

For any $0<\alpha <1$ and $x\in {\text{Ω}}_X$, there exists a ($1-\alpha$)-level generalized highest posterior density credible set of the form ${\varphi }^{*}$ in (1) with $\gamma (\theta \mid x)=\gamma (x)$. Furthermore, if ${\varphi }^{\prime }$ is any smallest generalized credible set of level $1-\alpha$, then it has the form (1) a.s. $P_X$. That is,

$$P_{\text{Θ}\mid X}\{{\varphi }^{*}(\theta \mid x)\ne {\varphi }^{\prime }(\theta \mid x),{\pi }_{\text{Θ}\mid X}(\theta \mid x)\ne {\kappa }_{\alpha }(x)\mid X=x\}=0\text{a.s.}{P}_X.$$

Let Define $\rho (\kappa \mid x)=P_{\text{Θ}\mid X}\{{\pi }_{\text{Θ}\mid X}(\theta \mid x)>\kappa \mid x\}$. The next proposition shows that the generalized highest posterior density credible set reduces to the highest posterior density credible set when $\rho (\cdot \mid x)$ is continuous at ${\kappa }_{\alpha }$.

Proposition 3.1.

Let ${\kappa }_{\alpha }(x)=\text{sup}\{\kappa :P\{{\pi }_{\text{Θ}\mid X}(\theta \mid x)\ge \kappa \mid x\}\ge 1-\alpha \}$. If $\rho$ is continuous at ${\kappa }_{\alpha }(x)$, then the generalized highest posterior density credible set ${\varphi }^{*}$ in (1) reduces to

$${\varphi }^{*}(\theta \mid x)=\left\{\begin{array}{lll}1& \text{if}& {\pi }_{\text{Θ}\mid X}(\theta \mid x)\ge {\kappa }_{\alpha }(x)\\ 0& \text{if}& {\pi }_{\text{Θ}\mid X}(\theta \mid x)<{\kappa }_{\alpha }(x)\end{array}\right.$$

so that $\{\theta :{\varphi }^{*}(\theta \mid x)=1\}$ is the ($1-\alpha$)-level highest posterior density credible set.

4. Fair generalized highest posterior density credible set

The generalized highest posterior density credible set ${\varphi }^{*}$ in (1) is not unique, unless $P_{\text{Θ}\mid X}\{{\pi }_{\text{Θ}\mid X}(\theta \mid x)={\kappa }_{\alpha }(x)\mid x\}=0$. When $P_{\text{Θ}\mid X}\{{\pi }_{\text{Θ}\mid X}(\theta \mid x)={\kappa }_{\alpha }(x)\mid x\}>0$, any $\gamma (\theta \mid x)$ satisfying

$${\int }_{K(x)}\gamma (\theta \mid x){\pi }_{\text{Θ}\mid X}(\theta \mid x)d{\mu }_{\text{Θ}}(\theta )=1-\alpha -P_{\text{Θ}\mid X}\{{\pi }_{\text{Θ}\mid X}(\theta \mid x)>{\kappa }_{\alpha }(x)\mid x\}$$ (2)

is optimal in the sense of Theorem 3.2 where $K(x)=\{\theta :{\pi }_{\text{Θ}\mid X}(\theta \mid x)={\kappa }_{\alpha }(x)\}$. In other words, if we let

$$S=\{{\varphi }^{*}:{\varphi }^{*}\text{is of the form (1) satisfying (2)}\},$$ (3)

then every member of $S$ is the generalized highest posterior density credible set. For definiteness, we introduce the fair generalized highest posterior density credible set as follows.

Definition 4.1.

Let $S$ be the collection of all generalized highest posterior density credible set of level-($1-\alpha$) as defined in (3). We call ${\varphi }_f^{*}\in S$ the fair generalized highest posterior density credible set if

$$\text{var}\{{\varphi }_f^{*}(\theta \mid x)\}\le \text{var}\{{\varphi }^{*}(\theta \mid x)\}\text{for all}{\varphi }^{*}(\theta \mid x)\in S.$$

When there is more than one generalized highest posterior density credible set, it is desirable to make ${\varphi }^{*}(\theta \mid x)$ as similar as possible to give each $\theta$ a fair treatment. That is, we should minimize the variation of ${\varphi }^{*}(\theta \mid x)$ with respect to ${\pi }_{\text{Θ}\mid X}(\theta \mid x)$.

The above definition is needed only when $P_{\text{Θ}\mid X}\{{\pi }_{\text{Θ}\mid X}(\theta \mid x)={\kappa }_{\alpha }(x)\}>0$. When this probability is 0, it doesn’t matter how we define $\gamma (\theta \mid x)$. For example, we can take $\gamma (\theta \mid x)=0$ on the set $\{\theta :{\pi }_{\text{Θ}\mid X}(\theta \mid x)={\kappa }_{\alpha }(x)\}$. The following theorem characterizes the form of the fair generalized highest posterior density credible set.

Theorem 4.2.

Suppose $P_{\text{Θ}\mid X}\{{\pi }_{\text{Θ}\mid X}(\theta \mid x)={\kappa }_{\alpha }(x)\}>0$. Then the fair generalized highest posterior density credible set ${\varphi }_f^{*}$ is the member of $S$ with

$$\gamma (\theta \mid x)=\frac{1-\alpha -P_{\text{Θ}\mid X}\{{\pi }_{\text{Θ}\mid X}(\theta \mid x)>{\kappa }_{\alpha }(x)\mid x\}}{P_{\text{Θ}\mid X}\{{\pi }_{\text{Θ}\mid X}(\theta \mid x)={\kappa }_{\alpha }(x)\mid x\}}.$$

To compute the generalized highest posterior density credible set, imagine the posterior probabilities at $\left\{{\theta }_1,{\theta }_2,\dots \right\}$ as spikes whose heights correspond to the posterior probabilities. We simply collect the set of highest spikes so that adding another spike (or a few more spikes, if the next few spikes are equal in height) would make the total height exceed the credible level $1-\alpha$. We then put a weight on the next spike (or distribute the weight evenly among the next few spikes if they are equal in height), so that the posterior probabilities and the weighted posterior probabilities add up to $1-\alpha$.

5. Data application

We now apply the fair generalized highest posterior density credible set and the steering wheel plot, which can be found in the Supplementary Material, to speaker accent recognition data. It is available at the UCI Machine Learning Repository at https://archive.ics.uci.edu/ml/datasets/Speaker+Accent+Recognition#; see also [15]. There are 12 integer features and six different accents of English: Spanish, French, German, Italian, British, and American. The data has a sample size of 329. Speakers from six different countries read single English words. Twelve features for each subject are extracted from the soundtrack of no more than one second of reading a word using Mel-Frequency Cepstral Coefficients.

For better visual effect, we first perform sufficient dimension reduction to reduce the dimension of the predictors [16]. We apply sliced inverse regression [17] and directional regression [18] to find low-dimensional predictors with four dimensions. We then conduct quadratic discriminant analysis to classify the accents in the reduced space. The 95%-level fair generalized highest posterior density credible set and steering wheel plot (as described in the Supplementary Material) enable us to infer the classification uncertainty.

Figure 1 shows the distinction between the classification uncertainties of two different quadratic discriminant analysis classifiers. For example, considering the French accent, the spokes of the classifier with sliced inverse regression are French, American, Spanish, or British, whereas the spokes of the classifier with directional regression are French or American. We can infer that directional regression helps to reduce classification uncertainty when classifying French accents.

Figure 1.

Speaker accent data: the steering wheel plots (as described in the Supplementary Material) of the first two sufficient predictors. Spanish is red, French is green, German is blue, Italian is black, British is orange, and American is purple. To enhance clarity, five samples are randomly selected for each classified accent, resulting in a total of 30 samples. (a) sliced inverse regression predictors; (b) directional regression predictors.

6. Discussion

In this paper, we generalize the classical Bayesian credible set to achieve exact credible levels for discrete parameters. This fills a gap in Bayesian uncertainty quantification when the random parameter is supported on a finite or discrete set, as in classification problems such as linear or quadratic discriminant analysis.

As a referee pointed out, our construction has a connection with fuzzy Bayesian inference (see, for example, [12], [13], and [14]), and, in particular, with the idea of the membership function. In fact, our generalized credible set $\varphi (\theta |x)$ can be viewed as a membership function whose greater value (close to 1) represents closer proximity to the credible set. However, there are some important differences in the motivation, formulation, and theory of our approach from fuzzy Bayesian inference.

The original motivation of our construction of the function $\varphi (\theta |x)$ is the “randomized test” in the classical theory of hypothesis testing. See, for example, [11]. The randomized test generalizes the critical region (rejection region), which is equivalent to an indicator function, by a function that takes values in the interval [0, 1] (as opposed to the set {0, 1}). This was precisely to take care of the scenarios where the significance level $\alpha$ cannot be precisely attained. We transplant this idea to the Bayesian setting and apply it to discrete parameters.

However, beyond the membership function, the overlap between our problem and that of fuzzy Bayesian inference is quite small: we do not consider any fuzzy data, where $x$ is replaced by a fuzzy number, nor any fuzzy parameter, where $\theta$ is replaced by a fuzzy number. Our setting is completely that of classical Bayesian inference, except that we introduced the analogy of a randomized test, or membership function, to achieve the exact credible level.

Although [12] and [13] also developed the highest posterior density credible set, their purposes were to extend this concept to the context of fuzzy data and fuzzy parameters. However, our purpose in extending the highest posterior density credible set is simply to accommodate discrete parameters. More importantly, the main theoretical novelty of our paper is to establish the existence, optimality, and uniqueness of the generalized highest posterior density credible set by borrowing the argument of the Neyman-Pearson lemma. This key component, however, was absent in the mentioned earlier works.

Our optimal result in Theorem 3.2 can be stated through a Bayes rule in an appropriately formulated statistical decision, which will give further insight into our method. Consider the statistical decision problem with the following components.

1. Parameter space ${\text{Ω}}_{\text{Θ}}$: this is a discrete or finite set of parameter values (or class labels), say, ${\text{Ω}}_{\text{Θ}}=\left\{{\theta }_1,{\theta }_2,\dots \right\}$.

2. Action space ${\text{Ω}}_A$: this is the collection of all functions $\varphi :{\text{Ω}}_{\text{Θ}}\to [\mathrm{0,\; 1}]$ such that

   $$E[\varphi (\text{Θ})\mid X]=1-\alpha .$$

   That is, each member of ${\text{Ω}}_A$ is a generalized credible set of credible level 1 − $\alpha$.

3. Class of decision rules $𝒟$: this is the set of all mappings from ${\text{Ω}}_X$ to ${\text{Ω}}_A$; that is, a member of $𝒟$ is a function $x\mapsto \varphi (\cdot \mid x)$, where $\varphi (\cdot \mid x)$ is a member of ${\text{Ω}}_A$

4. Loss function: given a parameter value $\theta \in {\text{Ω}}_{\text{Θ}}$, and an action $\varphi \in {\text{Ω}}_A$, we define the loss function as

   $$L(\theta ,\varphi )={\int }_{{\text{Ω}}_{\text{Θ}}}\varphi (\theta )d{\mu }_{\text{Θ}}(\theta ).$$
5. Posterior Expected Loss: given a decision $x\mapsto \varphi (\cdot \mid x)$ in $𝒟$, the posterior expected loss is

   $${\int }_{{\text{Ω}}_{\text{Θ}}}\varphi (\theta \mid X)d{\mu }_{\text{Θ}}(\theta ).$$

6. Bayes Rule: Our Theorem 3.2 shows that the generalized highest posterior density credible set defined in Definition 3.1 is, in fact, the Bayes rule with respect to the loss $L(\theta ,\varphi )$, among the decision rules in $𝒟$.