Asymptotic Geometry of Multiple Hypothesis Testing

Abstract

We present a simple geometrical interpretation for the solution to the multiple hypothesis testing problem in the asymptotic limit. Under this interpretation, the optimal decision rule is a nearest neighbor classifier on the probability simplex.

I. Introduction

In the binary hypothesis testing problem with independent and identically distributed (i.i.d.) observations, it is well known that the error probability for the optimal decision rule decays with a constant exponential rate equal to the Chernoff distance between the two hypothesis distributions. The generalization to multiple hypothesis testing for i.i.d. observations was derived by Leang and Johnson [1], and extended for observations modeled as stationary ergodic processes by Schmid and O’Sullivan [2]. In this correspondence, we focus on the i.i.d. case. For M-ary hypothesis testing, the error probability decays exponentially with a rate equal to the minimum Chernoff distance between all distinct pairs of hypothesis distributions. In this correspondence, we describe a simple geometrical interpretation of this result, illustrated in Fig. 2. We first review the binary case.

Fig. 2.

M-ary hypothesis testing.

II. Binary Hypothesis Testing

Let π_i, i = 1, 2 be the prior probabilities for the hypotheses H₁ and H₂; and let Yⁿ = (Y₁, …., Y_n) be an i.i.d. sequence of observations from a finite alphabet $\mathcal{Y}$. We must decide between

$$H_1:Y^n~P_1$$

and

$$H_2:Y^n~P_2.$$

A. Optimal Decision Rule and Error Exponent

The following results are standard (see, e.g., [3, Ch. 12]). Letting π₁ = π₂ = 1/2, the optimal decision $\widehat{H}=\widehat{H}\left(Y^n\right)$ rule can be expressed in terms of the Kullback–Leibler (KL)-divergence as

$$\widehat{H}=\underset{i\in \{1,2\}}{\text{argmin\hspace{0.17em}}}D\left(P_{Y^n}\Vert P_i\right)$$ (1)

where $P_{Y}n$ is the type (empirical histogram) of Yⁿ. This decision rule partitions the probability simplex $\mathcal{P}$ over $\mathcal{Y}$ into two disjoint decision regions A₁, A₂. Denoting their respective complements $A_1^c=A_2,A_2^c=A_1$, an application of Sanov’s theorem shows that the error probabilities are

$$\begin{array}{l}{P}_1^n\left(A_1^c\right)\doteq 2^{-nD\left(P_1^{*}\Vert P_1\right)}\\ P_2^n\left(A_2^c\right)\doteq 2^{-nD\left(P_2^{*}\Vert P_2\right)}\end{array}$$ (2)

where

$$P_i^{*}\triangleq \underset{\mathit{p}\in A_i^c}{\text{argmin\hspace{0.17em}}}D\left(\mathit{p}\Vert P_i\right),i=1,2$$

and ≐ denotes “equality to first order in the exponent,” i.e., a_n ≐ b_n means ${\text{lim}}_{n\to \infty }\frac{1}{n}\text{log}\frac{a_n}{b_n}=0$ (see [3, p. 55]). For the total probability of error $P_e^n$ we have

$$\begin{gathered} P_e^n={\pi }_1{P}_1^n\left(A_1^c\right)+{\pi }_2{P}_2^n\left(A_2^c\right) \\ \doteq {\pi }_1{2}^{-nD\left(P_1^{*}\Vert P_1\right)}+{\pi }_2{2}^{-nD\left(P_2^{*}\Vert P_2\right)} \\ \doteq 2^{-n\text{\hspace{0.17em}\hspace{0.17em}min}\left\{D\left(P_1^{*}\Vert P_1\right),D\left(P_2^{*}\Vert P_2\right)\right\}} \\ \triangleq 2^{-nC\left(P_1,P_2\right)}. \end{gathered}$$ (3)

The exponent

$$C\left(P_1,P_2\right)=\text{min}\left\{D\left(P_1^{*}\Vert P_1\right),D\left(P_2^{*}\Vert P_2\right)\right\}$$

is the Chernoff distance between P₁ and P₂, usually written in the alternative but equivalent form

$$C\left(P_1,P_2\right)=\underset{\lambda \in [0,1]}{\text{min}}\text{log}\left(\sum _y{P}_1^{\lambda }(y)P_2^{\lambda -1}(y)\right).$$

B. Geometric Interpretation

Fig. 1 illustrates the preceding results for alphabet size $|\mathcal{Y}|=3$, with corresponding probability simplex

$$\mathcal{P}=\left\{\mathit{p}:p_1,p_2,p_3\ge 0,p_1+p_2+p_3=1\right\}.$$ (4)

In this illustration, each probability distribution $\mathit{p}\in \mathcal{P}$ is mapped to a point $\mathit{r}\in {ℝ}^2$ in an equilateral triangle with vertices at points $\mathit{r}=(x,y)\in {ℝ}^2:(0,1),(\sqrt{3}/2,-1/2)$, and $(-\sqrt{3}/2,-1/2)$, using the transformation r = Tp + b, where

$$\mathit{T}=\left(\begin{array}{cc}-1/\sqrt{3}& -1/3\\ 1/\sqrt{3}& -1/3\\ 0& 2/3\end{array}\right),\mathit{b}=\left(\begin{array}{c}1/3\\ 1/3\\ 1/3\end{array}\right).$$

Fig. 1.

Binary hypothesis testing.

Given an observation sequence Yⁿ, the decision rule (1) can be interpreted geometrically in terms of the following algorithm:

• Locate the observation’s type $P_{Y}n$ in the simplex $\mathcal{P}$;

• Compute the distance to each hypothesis distribution $D\left(P_{Y^n}\Vert P_1\right),D\left(P_{Y^n}\Vert P_2\right)$;

• Assign Yⁿ to the closest hypothesis.

$\widehat{H}$ thus divides $\mathcal{P}$ into two disjoint cells,¹ with centroids P₁ and P₂, and each point $\mathit{p}\in \mathcal{P}$ assigned to the nearest centroid, with “distance” measured in KL-divergence.

Next consider (2). Under hypothesis H₁ : Yⁿ ~ P₁, the probability of incorrectly deciding $Y^n~{\mathcal{P}}_2$, that is, $P_1^n\left(A_1^c\right)$, depends on the distance from P₁ to the closest distribution outside of A₁. This distribution $P_1^{*}$ can be expressed as a point along the geodesic in $\mathcal{P}$ joining P₁ and P₂

$$\mathit{p}(\lambda )=\frac{1}{Z(\lambda )}{P}_1^{1-\lambda }{P}_2^{\lambda },\lambda \in [0,1]$$ (5)

where exponentiation is defined pointwise (e.g., $\text{if}|\mathcal{Y}|=3$, then ${\mathit{p}}^{\lambda }=\left(p_1^{\lambda },p_2^{\lambda },p_3^{\lambda }\right)$. Conceptually, the value λ* for which $\mathit{p}\left({\lambda }^{*}\right)=P_1^{*}$ can be found by setting λ = 0, then continuously increasing λ until the point p(λ) reaches the border separating regions A₁ and A₂. Symmetric arguments apply for $P_2^{*}$ and, by symmetry, we must have $P_1^{*}=P_2^{*}$.

Finally, interpreting (3), the overall probability of error $P_e^n$ is determined by measuring the distance along the geodesic from either centroid P₁ or P₂ to its border. The shortest of the two paths asymptotically dominates the overall probability of error.

III. M-Ary Hypothesis Testing

Now consider the M-ary hypothesis testing problem. Let π_i be the prior probability for hypothesis H_i. We must decide among

$$H_i:Y^n~P_i,i=1,\dots ,M.$$

As above, we assume equal prior probabilities, π_i = 1/M, i = 1, ….,M.

An important special case is the following formulation of the general statistical pattern recognition problem [4]: Given a set of M template patterns $\mathcal{C}=\left\{x^n(1),\dots ,x^n(M)\right\}$, suppose Nature selects one of the pattern templates w ∈ {1, …,M} at random with probability p(w) = 1/M, w = 1, ….,M. The observation data Yⁿ is generated from the template xⁿ(w) according to $P_w=p\left(y^n|x^n(w)\right)={\prod }_{j=1}^{n}p\left(y_j|x_j(w)\right)$. In this case, the optimal hypothesis test $\widehat{w}$ is a classifier, i.e., a rule that infers the pattern class underlying the observation Yⁿ.

A. Optimal Decision Rule and Error Exponent

The natural generalizations of (1)–(3) to the M-ary case, derived by Leang and Johnson [1], are as follows. The optimal decision rule is

$$\widehat{H}=\underset{i\in \{1,\dots ,M\}}{\text{argmin\hspace{0.17em}}}D\left(P_{Y^n}\Vert P_i\right).$$ (6)

Denoting the optimal decision regions by A_i, and their complements by $A_i^c$, applying Sanov’s theorem yields for the error probabilities under each hypothesis H_i, i = 1, …,M

$$\begin{gathered} P_i^n\left(A_i^c\right)\doteq 2^{-nD\left(P_i^{*}\Vert P_i\right)} \\ {P}_i^{*}\triangleq \underset{\mathit{p}\in A_i^c}{\text{argmin\hspace{0.17em}}}D\left(\mathit{p}\Vert P_i\right). \end{gathered}$$ (7)

Hence, for the total probability of error $P_e^n$ we have

$$\begin{gathered} P_e^n=\sum _{i=1}^M{\pi }_i{P}_i^n\left(A_i^c\right) \\ \doteq \sum _{i=1}^M{\pi }_i{2}^{-nD\left(P_i^{*}\Vert P_i\right)} \\ \doteq 2^{-n{\text{\hspace{0.17em}min}}_{i}D\left(P_i^{*}\Vert P_i\right)} \\ =2^{-nC\left(P_w,P_{w^{\prime }}\right)} \end{gathered}$$ (8)

where

$$\left(w,w^{\prime }\right)=\underset{i\ne j}{\text{argmin\hspace{0.17em}}}C\left(P_i,P_j\right)$$

i.e., P_w, P_w′ is the closest pair of distributions, measured in Chernoff distance.

B. Geometric Interpretation

Now consider the geometric interpretation of the preceding results for M-ary hypothesis testing. Fig. 2 represents the M-ary generalization of Fig. 1.

Given an observation sequence Yⁿ, the optimal decision rule (6) can be interpreted geometrically in terms of the following algorithm:

• Locate the observation’s type $P_{Y}n$ in the simplex $\mathcal{P}$;

• Compute the distance to each hypothesis distribution $D\left(P_{Y}n\Vert P_i\right),i=1,\dots ,M$;

• Assign Yⁿ to the closest hypothesis.

The rule $\widehat{H}$ thus divides $\mathcal{P}$ into M cells, with centroids P_i, i = 1, …,M, with each point $p\in \mathcal{P}$ assigned to the nearest centroid.

These cells are convex: Given a cell A_i with centroid P_i and two points Q₁,Q₂ ∈ A_i, let Q = λQ₁ + (1 − λ)Q₂, λ ∈ [0, 1], let P_j ∈ A_j be any other centroid, and define

$$\begin{gathered} \Delta \left(Q_k\right)\triangleq D\left(Q_k\Vert P_i\right)-D\left(Q_k\Vert P_j\right) \\ =E_{Q_k}\text{log}\frac{P_j}{P_i},k=1,2. \end{gathered}$$

The condition Q₁,Q₂ ∈ A_i is equivalent to requiring Δ(Q_k) < 0, k = 1, 2. Hence, noting that Δ(Q_k) is linear in Q_k, we have

$$\Delta (Q)=\lambda \Delta \left(Q_1\right)+(1-\lambda )\Delta \left(Q_2\right)<0$$

whence Q ∈ A_i, establishing the convexity of A_i. A similar argument shows that the borders of the decision cells are composed of straight lines: Q₁, Q₂ are on the boundary between A_i, A_j, if and only if Δ(Q₁) = Δ(Q₂) = 0, hence, Δ(Q) = λΔ(Q₁) + (1 − λ)Δ(Q₂) = 0.

Turning to (7), under each hypothesis H_i the corresponding probability of error $P_i\left(A_i^c\right)$ is again determined by the distance from the centroid P_i of A_i to the nearest point outside A_i. However, in the M-ary case, each complement $A_i^c$ consists of M − 1 other decision regions, and $P_1^{*}$ lies on the geodesic joining P_i with the closest neighboring region to A_i. Note that, unlike the binary case, for two neighboring cells A_i, A_j, it is not necessarily the case that $P_i^{*}=P_j^{*}$, although this is the case when geodesics intersect a shared border.

Finally, consider (8). To determine the overall probability of error $P_e^n$, we compare the geodesic distance from each centroid to its nearest border, $D\left(P_i^{*}\Vert P_i\right)$. The overall probability of error is dominated by the shortest such path, of length D(P_w‖P_w′).

IV. Conclusion

We have described a simple, easily remembered geometrical interpretation of the results of [1] for the multiple hypothesis testing problem in the asymptotic regime. Under this interpretation, the optimal decision rule is a nearest neighbor classifier, with “distance” measured in terms of the KL-divergence between the observation data’s type and each hypothesis distribution. That is, the optimal decision rule splits the probability simplex into M cells, and assigns each observation to the nearest centroid. The error probability under each hypothesis is determined by the geodesic distance from the centroid of its cell to the nearest cell border. The overall probability of error is determined by the smallest such distance.