A discussion on significance indices for contingency tables under small sample sizes

Abstract

Hypothesis testing in contingency tables is usually based on asymptotic results, thereby restricting its proper use to large samples. To study these tests in small samples, we consider the likelihood ratio test (LRT) and define an accurate index for the celebrated hypotheses of homogeneity, independence, and Hardy-Weinberg equilibrium. The aim is to understand the use of the asymptotic results of the frequentist Likelihood Ratio Test and the Bayesian FBST (Full Bayesian Significance Test) under small-sample scenarios. The proposed exact LRT p-value is used as a benchmark to understand the other indices. We perform analysis in different scenarios, considering different sample sizes and different table dimensions. The conditional Fisher’s exact test for 2 × 2 tables and the Barnard’s exact test are also discussed. The main message of this paper is that all indices have very similar behavior, except for Fisher and Barnard tests that has a discrete behavior. The most powerful test was the asymptotic p-value from the likelihood ratio test, suggesting that is a good alternative for small sample sizes.

Introduction

We discuss indices for homogeneity, independence, and Hardy-Weinberg equilibrium hypotheses [1, 2] in contingency tables. We propose an exact evaluation of the Likelihood Ratio Test (LRT) as a benchmark significance index. Based on the work of [3], its idea is to evaluate the probability distribution of all possible tables on the sample space under the null hypothesis. Once the distribution for sampling contingency tables under the hypothesis is known, we are able to compute the exact distribution of the Likelihood Ratio Test (LRT) statistics. The main difficulty for this procedure is that it is computationally time-consuming, being only feasible for small sample sizes and/or for tables of small dimension.

The exact LRT p-value presented as a way to do exact inference. The aim is to compare the behavior of the frequentist LRT asymptotic p-value [4], the exact LRT p-value, the Fisher’s exact test p-value [5], the Chi-Square test asymptotic p-value [6, 7] and the Barnard’s exact test p-value [8–11]. These frequentist indices are also compared to the e-value from the Full Bayesian Significance Test (FBST) [12, 13]. It was considered the asymptotic e-value and its approximation (based on a Markov Chain Monte Carlo procedure) of the exact e-value. The choice of adding a Bayesian index to the comparison study originates from the known asymptotic relationship between the LRT and the FBST [14]. Moreover, the FBST and its e-value can be viewed as a Bayesian p-value counterpart, and therefore it is interesting to understand this Bayesian method when compared to frequentist methods. It is important to point out that we are mainly interested in the values of the indices, not in the acceptance or rejection of the hypothesis; that is, our focus is on the significance test, which consists of the evaluation of the p-(e-)values. In an applied setting, the researcher can, based on the indices, make his/her decision about his/her application. We are not interested in comparing the values of the indices with some fixed significance value (generally 5%) to decide the if the hypothesis should be accepted or rejected. With this goal in mind, all significance indices considered here are in agreement with the ASA’s statement on significance indices [15].

From a historical perspective, hypothesis testing has been the most widely used statistical tool in many fields of science [16–18]. For categorical data, [19] discusses some exact procedures to perform inference and [20] presents methodological procedures for hypothesis testing for contingency tables. Tests for homogeneity hypothesis in contingency tables have been compared by [21], who compared the conditional and unconditional, and by [22], who compares, under an asymptotic perspective, two tests for equality of two proportions considering Goodman’s Y² and χ² statistics. Regarding tests for the independence of two classifiers in contingency tables, [23] presents an algorithm for finding the exact permutation significance level for r × c contingency tables. [24], studies a simple way to compare two correlated proportions. More recently, [25] presents the exact likelihood ratio test for equality of two normal populations, and [26] discuss exact unconditional tests for homogeneity hypothesis in 2 × 2 tables.

One important aspect that differentiates the tests procedures is how each one deals with the elimination of the nuisance parameter. Basu [27] lists several methods but focuses on marginalization and conditioning. He defines marginalization as every procedure that replaces the observed sample x by the observed value of a suitable statistic T(x) = t. Therefore, instead of working with the original experiment $ℰ$ and data x, one should use the marginal experiment ${ℰ}_T$ and the recorded value T(x) since the marginal statistical model would depend only on the parameter of interest. To justify these procedures, Basu adds that researchers usually recur to invariance or partial sufficiency arguments.

By conditioning, Basu defines methods of elimination that also consist of choosing a suitable statistic, but such that the conditional distribution of the observed sample, x, given the observed value of the statistic depends on the full parameter space only through the parameter of interest. Another commonly used approach that Basu describes is the one he calls maximization. In this case the nuisance parameter is eliminated from the risk function by some sort of maximization (or minimax) principle or directly from the likelihood, usually maximizing it with respect the nuisance parameters.

A final important strategy mentioned by Basu is the one he called Bayesian solution. In this case, one should derive the full posterior and integrate out the nuisance parameters, obtaining the posterior marginal distribution necessary to perform the required statistical inference. It is important to point out that the FBST does not follow this Bayesian strategy, since its evidence value is computed considering the full posterior. The proposed exact LRT p-value is based on the idea of integrating out the nuisance parameter, which is in some way related to Basu’s Bayesian solution [26]. The methods for elimination of nuisance parameters, maximization and Bayesian solution can be considered as unconditional methods.

The Likelihood Ratio Test (LRT) asymptotic p-value [28], the Chi-Square test asymptotic p-value [29], Fisher’s homogeneity exact test [29, 30], Barnard’s exact test [8], and the Full Bayesian Significance Test (FBST) asymptotic and exact e-value [12, 13] are presented in detail for the case of 2 × 2 contingency tables considering homogeneity hypothesis (Section 1.1). The theoretical results for homogeneity and independence hypotheses for tables of any dimension and Hardy-Weinberg equilibrium hypothesis are presented in sections 1.2, 1.3 and 1.4.

We study the relationship between indices in Section 2.1. [14] perform a similar study, however they consider continuous random variables using the e-value and the LRT p-value and show that these indices share an asymptotic relationship. In our case, the asymptotic LRT p-value, the exact LRT p-value and the Chi-Square p-value have similar behavior, including in small sample size scenarios. Both Fisher’s exact test and Barnard’s exact test have a discrete behavior for their p-values, being more clear for the Barnard’s exact test p-value. All tests are unconditional tests, except for the Fisher one, that is a conditional test. It is important to draw attention to the fact that the present results are not based on a simulation study, we compute the indices for all possible tables in the sample space.

In addition to our focus on the study of significance indices, we also provide, for the frequentist indices, a study of the power functions to compare the tests considering the homogeneity hypothesis (2 × 2 tables) and Hardy-Weinberg equilibrium hypothesis (Section 2.2). The Fisher’s exact test was the least powerful, followed by the Barnard’s exact test, Chi-Square test, the exact LRT and the asymptotic LRT, the most powerful one. We did not evaluate the power function for the FBST; firstly, because it is not the aim of the Bayesian paradigm, and secondly, to do so, it would be necessary to define a decision rule for the FBST, which is not in the scope of this paper. We also note that, under the hull hypothesis, considering the significance level 5%, all frequentist indices achieved 5% rejection as expected.

1 Methods

1.1 Homogeneity test for 2 × 2 contingency tables

Let X₁ and X₂ be two random variables, representing the rows (1 and 2) of Table 1, x₁₁ and x₂₁ being their observed values, and n_1⋅ and n_2⋅ fixed sample sizes. Consider the distributions of X₁ as Binomial(n_1⋅, θ₁₁) and X₂ a Binomial(n_2⋅, θ₂₁) for describing the chances of a subject belong to category (column) C₁ in two distinct populations. Both populations are partitioned into two categories (columns) C₁ and C₂ and the objective is to test homogeneity among the two unknown population frequencies, H: θ₁₁ = θ₂₁ = θ. This hypothesis is geometrically represented by a diagonal line of the unit square.

Table 1. Contingency table 2 × 2.

row\column	1	2	total
1	x11	x12	n1⋅
2	x21	x22	n2⋅

n_i⋅ = x_i1 + x_i2, i = 1, 2.

The likelihood function is specified by

$$\begin{array}{c}\hfill L({\theta }_{11},{\theta }_{21}\mid x_{11},x_{21},n_{1·},n_{2·})=\frac{n_{1·}!n_{2·}!}{x_{11}!x_{21}!x_{12}!x_{22}!}{\theta }_{11}^{x_{11}}{\theta }_{21}^{x_{21}}{(1-{\theta }_{11})}^{x_{12}}{(1-{\theta }_{21})}^{x_{22}},\end{array}$$ (1)

where 0 ≤ θ_i1 ≤ 1, i = 1, 2. Under H, the likelihood function simplifies to

$$\begin{array}{c}\hfill L(\theta \mid x_{11},x_{21},n_{1·},n_{2·},\mathit{H})=\frac{n_{1·}!n_{2·}!}{x_{11}!x_{21}!x_{12}!x_{22}!}{\theta }^{x_{11}+x_{21}}{(1-\theta )}^{x_{12}+x_{22}},0\le \theta \le 1,\end{array}$$ (2)

and the LRT test statistics is:

$$\begin{array}{lll}\lambda \left(x_{11},x_{21}\right)\hfill & =\hfill & \frac{\underset{\theta \in {\Theta }_{\mathit{H}}}{\text{sup}}L\left({\theta }_{11},{\theta }_{21}\mid x_{11},x_{21},n_{1·},n_{2·}\right)}{\underset{\theta \in \Theta }{\text{sup}}L\left({\theta }_{11},{\theta }_{21}\mid x_{11},x_{21},n_{1·},n_{2·}\right)}\hfill \\ \hfill & =\hfill & \frac{{\left(\frac{x_{11}+x_{21}}{n_{1·}+n_{2·}}\right)}^{x_{11}+x_{21}}{\left(\frac{x_{12}+x_{22}}{n_{1·}+n_{2·}}\right)}^{x_{12}+x_{22}}}{{\left(\frac{x_{11}}{n_{1·}}\right)}^{x_{11}}{\left(\frac{x_{12}}{n_{1·}}\right)}^{x_{12}}{\left(\frac{x_{21}}{n_{2·}}\right)}^{x_{21}}{\left(\frac{x_{22}}{n_{2·}}\right)}^{x_{22}}},\hfill \end{array}$$ (3)

in which Θ_H is the parametric set defined by the hypothesis.

• • Exact LRT p-value:

To define this p-value, we use the predictive distributions of X₁ and X₂ before any data were observed. The proposed p-value is an alternative way to calculate an exact p-value for the LRT. The goal is to find a distribution for the contingency table under H that is not a function on θ. We consider θ a nuisance parameter in the likelihood function in (2) and integrate it over θ in order to eliminate it, as suggested by [27]. The idea is to incorporate the concept of the Bayesian solution nuisance parameter elimination approach but in a frequentist setting, which means using the likelihood function instead of a posterior distribution. That is,

$$\begin{array}{ll}h(x_{11},x_{21})\hfill & ={\displaystyle {\int }_0^{1}L}(\theta \mid x_{11},x_{21},n_{1\cdot },n_{2\cdot },\mathit{H})\text{d}\theta \hfill \\ \hfill & =\frac{n_{1\cdot }!n_{2\cdot }!}{x_{11}!x_{21}!x_{12}!x_{22}!}{\displaystyle {\int }_0^1{\theta }^{x_{11}+x_{21}}}{(1-\theta )}^{x_{12}+x_{22}}\text{d}\theta \hfill \\ \hfill & =\left(\begin{array}{c}{n}_{1\cdot }\\ x_{11}\end{array}\right)\left(\begin{array}{c}{n}_{2\cdot }\\ x_{21}\end{array}\right)\frac{(x_{11}+x_{21})!(x_{12}+x_{22})!}{(n_{1\cdot }+n_{2\cdot }+1)!}\hfill \\ \hfill & =\frac{\left(\begin{array}{c}{n}_{1\cdot }\\ x_{11}\end{array}\right)\left(\begin{array}{c}{n}_{2\cdot }\\ x_{21}\end{array}\right)}{\left(\begin{array}{c}{n}_{1\cdot }+n_{2\cdot }\\ x_{11}+x_{21}\end{array}\right)}\frac{1}{(n_{1\cdot }+n_{2\cdot }+1)}.\hfill \end{array}$$ (4)

To obtain the probability function Pr(X₁ = x₁₁, X₂ = x₂₁ ∣ H), one needs to find a normalization constant.

$$\begin{array}{c}\hfill \text{Pr}(X_1=x_{11},X_2=x_{21}\mid \mathit{H})=\frac{h(x_{11},x_{21})}{{\displaystyle \sum _{i=0}^{n_{1·}}}{\displaystyle \sum _{j=0}^{n_{2·}}}h(i,j)}.\end{array}$$ (5)

Note that to calculate (5), we evaluate h(⋅, ⋅) for all possible tables. In the case of a homogeneity hypothesis for 2 × 2 contingency tables, ${\displaystyle \sum _{i=0}^{n_{1\cdot }}}{\displaystyle \sum _{j=0}^{n_{2\cdot }}}h(i,j)=1$. We present the table’s probability in terms of this sum to obtain a general formula for all hypotheses and table dimensions considered here, since in other scenarios this quantity does not sum up to 1 (for example, the sum of h for all possible 2 × 2 tables considering independence hypothesis with n = 2 is 2304). The exact p-value calculation follows directly from the test statistic distribution:

$$\begin{array}{ccc}\hfill \text{p-value}& =& \text{Pr}(\lambda (X_1,X_2)\le \lambda (x_{11},x_{21})\mid \mathit{H})\hfill \\ & =& \sum _{(i,j)\in \mathit{R}}\text{Pr}(X_1=i,X_2=j\mid \mathit{H}),\hfill \end{array}$$ (6)

in which R is the set of all pairs (i, j) such that λ(i, j) ≤ λ(x₁₁, x₂₁), and λ(x₁₁, x₂₁) is the observed test statistic, as in (3).

• • Barnard’s Exact Test:

Consider that n_1⋅ and n_2⋅ are fixed in Table 1. The random variables X₁ and X₂ are independent Binomial distribution with parameters θ₁₁ and θ₂₁. The probability of a sample {x₁₁, x₂₁} be drawn is

$$\begin{array}{c}\hfill \text{Pr}(X_1=x_{11},X_2=x_{21})=\frac{n_{1·}!}{x_{11}!x_{12}!}{\theta }_{11}^{x_{11}}{(1-{\theta }_{11})}^{x_{12}}\frac{n_{2·}!}{x_{21}!x_{22}!}{\theta }_{21}^{x_{21}}{(1-{\theta }_{21})}^{x_{22}},\end{array}$$ (7)

and, under hypothesis H,

$$\begin{array}{c}\hfill \text{Pr}(X_1=x_{11},X_2=x_{21}\mid \mathit{H})=\frac{n_{1·}!n_{2·}!}{x_{11}!x_{12}!x_{21}!x_{22}!}{\theta }^{x_{11}+x_{12}}{(1-\theta )}^{x_{12}+x_{22}}.\end{array}$$ (8)

We define the critical region as R = {λ(X₁, X₂) ≤ λ(x₁₁, x₂₁)}, then the Barnard’s exact p-value is obtained by

$$\begin{array}{c}\hfill \text{p-value}=\underset{0\le \theta \le 1}{\text{max}}\sum _{\mathit{R}}\frac{n_{1·}!n_{2·}!}{x_{11}!x_{12}!x_{21}!x_{22}!}{\theta }^{x_{11}+x_{12}}{(1-\theta )}^{x_{12}+x_{22}}.\end{array}$$ (9)

That is, the Barnard’s exact test consider the p-values for all possible points of the parameter space under H, and takes the maximum p-value. In this test, the chosen approach for nuisance parameter elimination among the ones presented by Basu is maximization.

• • Full Bayesian Significance Test:

The Bayesian approach considered is based on the FBST (Full Bayesian Significance Test) [12, 13].

Definition 1 Let π(θ ∣ x) be the posterior density function of θ given the observed sample and $T\left(\mathit{x}\right)=\{\theta \in \Theta :\pi (\theta \mid \mathit{x})\ge {\text{sup}}_{\theta \in {\Theta }_{\mathit{H}}}\pi (\theta \mid \mathit{x}\left)\right\}$. The supporting evidence measure for the hypothesis θ ∈ Θ_H is defined as Ev(Θ_H, x) = 1 − Pr(θ ∈ T(x) ∣ x).

Consider that, a priori, θ₁₁ and θ₂₁ are independent and both follow a Uniform(0, 1) distribution. The choice of uniforms priors is to avoid a subjective prior to have a fair comparison with frequentist indices. Recall that X₁ and X₂ given θ₁₁ and θ₂₁ are Binomial distributed. Hence, the posterior distributions for θ₁₁ and θ₂₁ are independent Beta(x₁₁ + 1, n_1⋅ − x₁₁ + 1) and Beta(x₂₁ + 1, n_2⋅ − x₂₁ + 1). Under the hypothesis H, the posterior distribution is

$$\begin{array}{c}\hfill \pi (\theta \mid x_{11},x_{21},n_{1·},n_{2·},\mathit{H})=\frac{{\theta }^{x_{11}+x_{21}}{(1-\theta )}^{x_{12}+x_{22}}}{\mathcal{B}(x_{11}+1,x_{12}+1)\mathcal{B}(x_{21}+1,x_{22}+1)}\end{array}$$ (10)

and by maximizing it in θ we obtain sup_θ∈(0,1) π(θ ∣ x₁₁, x₂₁, n_1⋅, n_2⋅, H), where $ℬ(\cdot ,\cdot )$ is the Beta function. Since x₁₁, x₂₁, n_1⋅ and n_2⋅ are integers,

$$\begin{array}{l}\pi \left(\theta \mid x_{11},x_{21},n_{1·},n_{2·},\mathit{H}\right)=\\ \left(\begin{array}{c}{n}_{1·}\\ x_{11}\end{array}\right)\left(\begin{array}{c}{n}_{2·}\\ x_{21}\end{array}\right)\left(n_{1·}+1\right)\left(n_{2·}+1\right){\theta }^{x_{11}+x_{21}}{\left(1-\theta \right)}^{x_{12}+x_{22}},\end{array}$$ (11)

$$\begin{array}{l}\underset{\theta \in \left(0,1\right)}{\text{sup}}\pi \left(\theta \mid x_{11},x_{21},n_{1·},n_{2·},\mathit{H}\right)=\\ \frac{\left(n_{1·}+1\right)!\left(n_{2·}+1\right)!}{x_{11}!x_{21}!x_{12}!x_{22}!}{\left(\frac{x_{11}+x_{21}}{n_{1·}+n_{2·}}\right)}^{x_{11}+x_{21}}{\left(\frac{x_{12}+x_{22}}{n_{1·}+n_{2·}}\right)}^{x_{12}+x_{22}},\end{array}$$ (12)

the hypothesis’ tangent set, T, is

$$\begin{array}{c}T(x_{11},x_{21},n_{1·},n_{2·})=\{({\theta }_{11},{\theta }_{21})\in (0,1)\times (0,1):\hfill \\ \hfill \pi ({\theta }_{11},{\theta }_{21}\mid x_{11},x_{21},n_{1·},n_{2·})\ge \underset{\theta \in (0,1)}{\text{sup}}\pi (\theta \mid x_{11},x_{21},n_{1·},n_{2·},\mathit{H})\},\end{array}$$ (13)

and

$$\begin{array}{c}\hfill \text{e-value}=1-\text{Pr}[\theta \in T(x_{11},x_{21},n_{1·},n_{2·})].\end{array}$$ (14)

To calculate the approximate e-value, we use the following algorithm:

1. A random sample of size k is generated from posterior distribution of θ₁₁, θ₂₁, obtaining $\{{\theta }_{x_{11}1},{\theta }_{x_{21}1}\},\dots ,\{{\theta }_{x_{11}k},{\theta }_{x_{21}k}\}$ .

2. The e-value is calculated by

   $$\begin{array}{c}\hfill 1-\frac{1}{k}\sum _{i=1}^{k}I(\pi ({\theta }_{x_{11}i},{\theta }_{x_{21}i}\mid x_{11},x_{21},n_{1·},n_{2·})\ge \underset{\theta \in (0,1)}{\text{sup}}\pi (\theta \mid x_{11},x_{21},n_{1·},n_{2·})),\end{array}$$

   in which I(A) is the indicator function of set A.

• • Other indices:

For the LRT, the statistic −2 ln[λ(X₁, X₂)] has asymptotically a chi-square distribution with 1 degree of freedom, which is dim(Θ) − dim(Θ_H) [28]. The FBST uses the same statistic, however its asymptotic distribution is a chi-square with 2 degrees of freedom [13], which is dim(Θ). For the chi-square test and the Fisher’s exact test for homogeneity see [29].

1.2 Homogeneity hypothesis for ℓ × c contingency tables

Let X_i, i = 1, …, ℓ, be random variables that are represented by the rows of Table 2 and n_1⋅, n_2⋅, …, n_ℓ⋅ are known constants.

Table 2. Contingency table ℓ × c.

row\column	1	2	⋯	c	total
1	x11	x12		x1c	n1⋅
2	x21	x22		x2c	n2⋅
⋮			⋱	⋮	⋮
ℓ	xℓ1	xℓ2	⋯	xℓc	nℓ⋅
total	n⋅1	n⋅2	⋯	n⋅c	n⋅⋅

$n_{i\cdot }={\sum }_{j=1}^c{x}_{ij}$, i = 1, …, ℓ;

$n_{\cdot j}={\sum }_{i=1}^{\ell }{x}_{ij}$, j = 1, …, c; and

$n_{\cdot \cdot }={\sum }_{i=1}^{\ell }{\sum }_{j=1}^c{x}_{ij}$.

Assuming that X_i, i = 1, …, ℓ, follows a Multinomial(n_i⋅, θ_i1, …, θ_ic) distribution, we are interested in testing if their distributions are homogeneous with respect to categories (columns) C_j, j = 1, …, c. That is,

$$\begin{array}{c}\hfill \mathit{H}:\{\begin{array}{ccc}\hfill {\theta }_1& =& {\theta }_{11}={\theta }_{21}=\cdots ={\theta }_{\ell 1},\hfill \\ \hfill {\theta }_2& =& {\theta }_{12}={\theta }_{22}=\cdots ={\theta }_{\ell 2},\hfill \\ & ⋮& \\ \hfill {\theta }_{c-1}& =& {\theta }_{1(c-1)}={\theta }_{2(c-1)}=\cdots ={\theta }_{\ell (c-1)},\hfill \end{array}\end{array}$$

in which ${\theta }_c=1-{\sum }_{k=1}^{c-1}{\theta }_k$, 0 ≤ θ_k ≤ 1, ∀k = 1, …, c.

Let x be all observed values presented in Table 2 and θ all the parameters. The likelihood function is

$$\begin{array}{c}\hfill L(\mathit{\theta }\mid \mathit{x})=[\prod _{i=1}^{\ell }{n}_{i·}!/\prod _{i=1}^{\ell }\prod _{j=1}^c{x}_{ij}!]\prod _{i=1}^{\ell }\prod _{j=1}^c{\theta }_{ij}^{x_{ij}},\end{array}$$ (15)

and under the hypothesis H,

$$\begin{array}{c}\hfill L(\mathit{\theta }\mid \mathit{x},\mathit{H})=[\prod _{i=1}^{\ell }{n}_{i·}!/\prod _{i=1}^{\ell }\prod _{j=1}^c{x}_{ij}!]\prod _{j=1}^c{\theta }_j^{n_{·j}}.\end{array}$$ (16)

The LRT λ statistic is

$$\begin{array}{c}\lambda \left(\mathit{x}\right)={\displaystyle \prod _{j=1}^c}{\left(\frac{n_{·j}}{n_{··}}\right)}^{n_{·j}}/{\displaystyle \prod _{i=1}^{\mathcal{l}}}{\displaystyle \prod _{j=1}^c}{\left(\frac{x_{ij}}{n_{i·}}\right)}^{x_{ij}}.\end{array}$$ (17)

• • Exact LRT p-value:

To obtain the exact LRT p-value, we need the function h(x). In this scenario,

$$\begin{array}{c}\hfill h\left(\mathit{x}\right)=\prod _{i=1}^{\ell }{n}_{i·}!\prod _{j=1}^c{n}_{·j}!/\left[\right(\prod _{i=1}^{\ell }\prod _{j=1}^c{x}_{ij}!)(n_{··}+c-1)!],\end{array}$$ (18)

and the p-value’s calculation follows as in Subsection 1.1.

• • FBST:

Assuming a Dirichlet(1, 1, …, 1) prior for {θ_i1, …, θ_ic}, and since X_i follows a Multinomial(n_i, θ_i1, …, θ_ic) distribution, then the posterior distribution is a Dirichlet(x_i1 + 1, x_i2 + 1, …, x_ic + 1), i = 1, …, ℓ.

In this setting,

$$\begin{array}{c}\underset{\mathit{\theta }\in {\Theta }_{\mathit{H}}}{\text{sup}}\pi \left(\mathit{\theta }\mid \mathit{x}\right)={\displaystyle \prod _{i=1}^{\mathcal{l}}}\left(n_{i·}+c-1\right)!{\displaystyle \prod _{j=1}^c}{\left(\frac{n_{·j}}{n_{··}}\right)}^{n_{·j}}/{\displaystyle \prod _{i=1}^{\mathcal{l}}}{\displaystyle \prod _{j=1}^c}{x}_{ij}!,\end{array}$$ (19)

and we can obtain the e-value from Definition 1.

• • Other indices:

Both asymptotic LRT p-value and asymptotic e-value are calculated as Pr[−2 ln(λ(X)) ≤ −2 ln(λ(x))], but while the LRT considers that this statistic follows a $X^2$ distribution with (ℓ − 1)(c − 1) degrees of freedom, the FBST considers that it follows a $X^2$ distribution with ℓ(c − 1) degrees of freedom. The Chi-Square homogeneity test is also obtained.

1.3 Independence hypothesis for ℓ × c contingency tables

Consider that θ_ij is the probability of observing a sample in the cell at row i and column j, θ_i⋅ is the probability of observing a sample in row i, θ_⋅j is the probability of observing a sample in column j, 0 ≤ θ_ij ≤ 1, 0 ≤ θ_i⋅ ≤ 1, 0 ≤ θ_⋅j ≤ 1, i = 1, …, ℓ, j = 1, …, c, ${\sum }_{i=1}^{\ell }{\sum }_{j=1}^c{\theta }_{ij}=1$, ${\sum }_{i=1}^{\ell }{\theta }_{i\cdot }=1$, and ${\sum }_{j=1}^c{\theta }_{\cdot j}=1$.

For the independence hypothesis, our interest is to test H: θ_ij = θ_i⋅ × θ_⋅j, ∀i, j. For the case of 2 × 2 table, the independence hypothesis is geometrically represented as Fig 1.

Fig 1. Geometric representation of the independence hypothesis (gray surface) for 2 × 2 tables.

The parametric space is the three-dimensional simplex (regular tetrahedron).

Considering that n_⋅⋅ is known, we assume that the outcomes of Table 2 follow a Multinomial(n.., θ) distribution, θ = {θ₁₁, …, θ_1(c−1), …, θ_ℓ1, …, θ_ℓ(c−1)}, and ${\theta }_{ic}=1-{\sum }_{j=1}^{c-1}{\theta }_{ij}$, i = 1, …, ℓ. The likelihood function is

$$\begin{array}{c}\hfill L(\mathit{\theta }\mid \mathit{x})=[n_{··}!/\prod _{i=1}^{\ell }\prod _{j=1}^c{x}_{ij}!]\prod _{i=1}^{\ell }\prod _{j=1}^c{\theta }_{ij}^{x_{ij}}.\end{array}$$ (20)

The likelihood function under H is

$$\begin{array}{c}\hfill L(\mathit{\theta }\mid \mathit{x},\mathit{H})=[n_{··}!/\prod _{i=1}^{\ell }\prod _{j=1}^c{x}_{ij}!]\prod _{i=1}^{\ell }{\theta }_{i·}^{n_{i·}}\prod _{j=1}^c{\theta }_{·j}^{n_{·j}},\end{array}$$ (21)

and the LRT λ statistic is

$$\begin{array}{c}\lambda \left(\mathit{x}\right)={\displaystyle \prod _{i=1}^{\mathcal{l}}}{\left(\frac{n_{i·}}{n_{··}}\right)}^{n_{i·}}{\displaystyle \prod _{j=1}^c}{\left(\frac{n_{·j}}{n_{··}}\right)}^{n_{·j}}/{\displaystyle \prod _{i=1}^{\mathcal{l}}}{\displaystyle \prod _{j=1}^c}{\left(\frac{x_{ij}}{n_{··}}\right)}^{x_{ij}}.\end{array}$$ (22)

• • Exact LRT p-value:

As shown in Subsection 1.1, this p-value is obtained the same way but with a different h(x). In this case,

$$\begin{array}{c}\hfill h\left(\mathit{x}\right)=n_{··}!(n_{··}+\ell )!(n_{··}+c)!/[\prod _{i=1}^{\ell }\prod _{j=1}^c{x}_{ij}!\prod _{i=1}^{\ell }{n}_{i·}!\prod _{j=1}^c{n}_{·j}!].\end{array}$$ (23)

• • FBST:

Assuming a Dirichlet(1, …, 1) as prior distribution for θ and that the outcomes of Table 2 follow a Multinomial(n, θ₁₁, …, θ_1c, …, θ_ℓ1, …, θ_ℓc) distribution, then the posterior distribution is a Dirichlet(x₁₁ + 1, …, x_1c + 1, …, x_ℓ1 + 1, …, x_ℓc1 + 1). The e-value is obtained from Definition 1 and

$$\begin{array}{c}\underset{\mathit{\theta }\in {\Theta }_{\mathit{H}}}{\text{sup}}\pi \left(\mathit{\theta }\mid \mathit{x}\right)=\left(n+\mathcal{l}c-1\right)!{\displaystyle \prod _{i=1}^{\mathcal{l}}}{\left(\frac{n_{i·}}{n}\right)}^{n_{i·}}{\displaystyle \prod _{j=1}^c}{\left(\frac{n_{·j}}{n}\right)}^{n_{·j}}/\left[{\displaystyle \prod _{i=1}^{\mathcal{l}}}{\displaystyle \prod _{j=1}^c}{x}_{ij}!\right].\end{array}$$ (24)

• • Other indices:

We obtained the asymptotic LRT p-value and e-value, considering that −2ln(λ(X)) follows a $X^2$ distribution with (ℓ − 1)(c − 1) and (ℓc − 1) degrees of freedom. We also obtained the p-value for the Chi-Square independence test.

1.4 Hardy-Weinberg equilibrium

An individual’s genotype is formed by a combination of alleles. If there are two possible alleles for one characteristic (say A and a), the possible genotypes are AA, Aa or aa. Considering a few premises true [31], the principle says that the allele probability in a population does not change from generation to generation. It is a fundamental principle for the Mendelian mating allelic model. If the probabilities of alleles are θ and 1 − θ, the expected genotype probabilities are (θ², 2θ(1 − θ), (1 − θ)²) 0 ≤ θ ≤ 1.

Considering the Hardy-Weinberg equilibrium, the aim is to verify if a population follows these genotypes proportions. Therefore, the equilibrium hypothesis is

$$\begin{array}{c}\hfill \mathit{H}:\{\begin{array}{ccc}\hfill {\theta }_1& =& {\theta }^2,\hfill \\ \hfill {\theta }_2& =& 2\theta (1-\theta ),\hfill \\ \hfill {\theta }_3& =& {(1-\theta )}^2,\hfill \end{array}\end{array}$$

in which θ₁, θ₂, θ₃ are the proportions of AA, Aa, and aa, respectively. This hypothesis is geometrically represented in Fig 2.

Fig 2. Geometric representation of the Hardy-Weinberg equilibrium hypothesis (black line), and the parametric space (gray shading).

Let X be a random vector. Table 3 represents the genotype frequencies for the population in question. Considering n known, we assume that X follows a Trinomial(n, θ₁, θ₂, θ₃) distribution. The likelihood function for this model is

$$\begin{array}{c}\hfill L(\mathit{\theta }\mid \mathit{x})=\left[\frac{n!}{x_1!x_2!x_3!}\right]{\theta }_1^{x_1}{\theta }_2^{x_2}{\theta }_3^{x_3},\end{array}$$ (25)

in which x = {x₁, x₂, x₃}, θ₁ + θ₂ + θ₃ = 1 and θ_i > 0, i = 1, 2, 3. Under the hypothesis H,

$$\begin{array}{c}\hfill L(\mathit{\theta }\mid \mathit{x},\mathit{H})=\left[\frac{n!}{x_1!x_2!x_3!}\right]2^{x_2}{\theta }^{2x_1+x_2}{(1-\theta )}^{2x_3+x_2},0\le \theta \le 1.\end{array}$$ (26)

Table 3. Genotype frequency.

	AA	Aa	aa	total
X	x1	x2	x3	n

n = x₁ + x₂ + x3.

The maximum likelihood estimator for θ under H is $\hat{\theta }=(2x_1+x_2)/\left(2n\right)$ and the LRT λ statistic is

$$\begin{array}{c}\hfill \lambda \left(\mathit{x}\right)=\frac{2^{x_2}{\widehat{\theta }}^{2x_1+x_2}{(1-\widehat{\theta })}^{2x_3+x_2}}{{\left(\frac{x_1}{n}\right)}^{x_1}{\left(\frac{x_2}{n}\right)}^{x_2}{\left(\frac{x_3}{n}\right)}^{x_3}}.\end{array}$$ (27)

• • Exact LRT p-value:

Calculations follow as for the other indices and in this scenario

$$\begin{array}{c}\hfill h\left(\mathit{x}\right)=\frac{n!2^{x_2}(2x_1+x_2)!(2x_3+x_2)!}{x_1!x_2!x_3!(2n+1)!}.\end{array}$$ (28)

• • Barnard’s Exact Test:

The critical region is R = {λ(X) ≤ λ(x)}, and the Barnard’s exact p-value is obtained by

$$\begin{array}{c}\hfill \text{p-value}=\underset{0\le \theta \le 1}{\text{max}}\sum _{\mathit{R}}\left[\frac{n!}{x_1!x_2!x_3!}\right]2^{x_2}{\theta }^{2x_1+x_2}{(1-\theta )}^{2x_3+x_2}.\end{array}$$ (29)

• • FBST:

Assuming a Dirichlet(1, 1, 1) prior for θ and that X follows a Trinomial(n, θ₁, θ₂, θ₃) distribution, the posterior distribution is θ ∣ x ~ Dirichlet(x₁ + 1, x₂ + 1, x₃ + 1). In this setting,

$$\begin{array}{c}\underset{\mathit{\theta }\in {\Theta }_{\mathit{H}}}{\text{sup}}\pi \left(\mathit{\theta }\mid \mathit{x}\right)=\frac{\left(n+2\right)!}{x_1!x_2!x_3!}{2}^{x_2}{\left(\frac{2x_1+x_2}{2n}\right)}^{2x_1+x_2}{\left(\frac{x_2+2x_3}{2n}\right)}^{x_2+2x_3}.\end{array}$$ (30)

• • Other indices:

Both asymptotic LRT p-value and asymptotic e-value are obtained, the p-value considering that −2 ln(λ(X)) follows a $X^2$ distribution with 1 degrees of freedom and the FBST considering that it follows a $X^2$ distribution with 2 degrees of freedom.

2 Results

2.1 Relations between the indices

In many practical situations, mainly in biological studies, asymptotic distributions are used to evaluate indices even for small samples. With that in mind, one of our interests is to understand how the use of asymptotic results for small sample size settings compares to the use of an exact index. Surprisingly, the values of exact and asymptotic indexes do not diverge considerably.

As our objective is to compare the indices, we consider different scenarios for each hypothesis. For each scenario, we evaluate the significance indices of all test procedures presented here. Note that this is not a simulation study; for each sample size, we evaluate the indices for all possible contingency tables of a fixed dimension and size. For example, considering homogeneity hypothesis in a 2 × 2 table with marginals (10, 10), there are 121 possible tables or considering independence hypothesis in a 2 × 3 table with marginal 15, there are 15504 possible tables. We evaluated the indices for all tables that fit into each specification. For the e-value computation, non-informative priors for the parameters are considered (that is, π(θ) ∝ 1). This way, no extra information is added besides the data, allowing fair comparisons between frequentist and Bayesian indices.

For each scenario, plots are drawn to illustrate differences between the indices’ values. The indices studied are the exact LRT p-value, asymptotic p-value for the LRT, asymptotic p-value for the chi-square test, e-value and asymptotic e-value. For the homogeneity hypothesis in 2 × 2 tables, Fisher and Barnard exact tests were also considered, and for Hardy-Weinberg equilibrium hypothesis the Barnard’s exact test was also obtained. We considered many different scenarios, however, since the aim is to understand the indices in small sample size, the scenarios presented here are in Table 4.

Table 4. Considered scenarios.

Setting	Hypothesis	Table	Sample sizes
1	Homogeneity	2 × 2	(30, 30)
2	Homogeneity	2 × 2	(100, 100)
3	Homogeneity	2 × 3	(30, 30)
4	Homogeneity	3 × 3	(15, 15, 15)
5	Independence	2 × 2	30
6	Independence	2 × 3	30
7	Independence	3 × 3	15
8	Independence	3 × 3	25
9	Hardy-Weinberg equilibrium	-	30
10	Hardy-Weinberg equilibrium	-	100

Figs 3, 4 and 5 illustrate the results of the discussion above. For all hypotheses, exact and asymptotic e-values are very similar for both large and small sample sizes. Looking into the frequentist indices, exact LRT p-values and asymptotic p-values, both LRT and Chi-Square, are also very similar to each other. The difference found between e-values when compared to asymptotic LRT p-value happens as a result of the way these indices are formulated: while e-values consider the full dimension of the parameter space, p-value consider the complementary dimension of the set corresponding to hypothesis H. This is expected from the asymptotic relationship between e-value and p-value from the LRT [13, 14]. Since the exact LRT p-value is directly related to the asymptotic LRT p-value, we observe the same behavior of the differences between e-values and asymptotic LRT p-value. Fisher’s exact test was only calculated for the homogeneity hypothesis in 2 × 2 tables, and Barnard’s exact test was calculated for the homogeneity hypothesis in 2 × 2 tables and for the Hardy-Weinberg equilibrium hypothesis. Both indices have a different behavior among the other indices considered. They have a discrete behavior, which is not surprising since Fisher’s exact test is a conditional test and Barnard’s exact test takes a maximization nuisance parameter elimination. Looking at the plots, their values do not form a continuous curve like the other indices’ values do, and its points are quite far from all the other indices.

Fig 3. Scaterplots for the significance indices of homogeneity hypothesis considering different sample sizes and different table dimensions.

The indices were evaluated for all possible samples in the sample space. The label in the top box of that column give the index in the x-axis, and the label in the left box of that row give the index in the y-axis. Each table dimesions and sample sizes are given in the sublabels.

Fig 4. Scaterplots for the significance indices of independence hypothesis considering different sample sizes and different table dimensions.

The indices were evaluated for all possible samples in the sample space. The label in the top box of that column give the index in the x-axis, and the label in the left box of that row give the index in the y-axis. Each table dimesions and sample sizes are given in the sublabels.

Fig 5. Scaterplots for the significance indices of Hardy-Weinberg hypothesis considering different sample sizes and different table dimensions.

The indices were evaluated for all possible samples in the sample space. The label in the top box of that column give the index in the x-axis, and the label in the left box of that row give the index in the y-axis. Each table dimesions and sample sizes are given in the sublabels.

2.2 Power function

Power functions are a useful tool to compare hypothesis tests. For all θ ∈ Θ, the power function provides the probability of rejecting the hypothesis for a given θ. In fact, we look for a test that does not reject the hypothesis for θ ∈ Θ_H and the further the θ value is from the hypothesis, the probability of rejection increases.

The power functions presented are the ones that we are able to represent in ℝ³, which are the power functions for the homogeneity hypothesis in 2 × 2 contingency tables and for the Hardy-Weinberg equilibrium hypothesis.

We used p-values less than 0.05 as a decision rule to reject the hypothesis. This choice is based on what is vastly used in most fields of science as a decision rule. In this case, Power(θ₁, θ₂) = P(reject H|(θ₁, θ₂) and Reject H if index ≤ 0.05.

We obtain the power function for all tests but the FBST. The FBST is a Bayesian significance test and in order to obtain a power function, one would need a decision rule. Since its construction differs from that of the p-values, we cannot use the same decision rule, and constructing a decision rule is not in the scope of this paper.

We used a Monte Carlo procedure to evaluate the power function of these tests. We consider a grid for the unit square with 100 × 100 points on the axes (θ₁, θ₂). For each point in the grid we generated 1000 tables. From these 1000 tables we evaluate the proportion of rejections, which is an approximation of the power function.

We plot pairs of power functions to illustrate and compare their shapes. For the homogeneity hypothesis in a table with marginals (10, 10), Fig 6 shows that Fisher’s exact test is less powerful than the Barnard’s exact test, the Barnard’s exact test is has similar power when compared with the Chi-square test, while the Chi-square is less powerful than the proposed exact LRT p-value, which is less powerful than the asymptotic p-value for the LRT. To have a clear picture, we plot the power functions from different tests against each other. Fig 7a consists of the power functions for tables with marginal equals to (10, 10). It shows that the use of the asymptotic p-value for the LRT results in a more powerful test than the other indices. When comparing the proposed exact p-value to other indices, it is more powerful than the Chi-square test and the Fisher’s exact test. Between the Chi-square and the Fisher’s exact test, the Chi-square test is more powerful.

Fig 6. Power function for homogeneity hypothesis in 2 × 2 contingency tables with n_1⋅ = n_2⋅ = 10.

Fig 7. Plots of power function values for the homogeneity test.

Each graph presents one index versus another, each dot representing a point in the considered parametric space (in this case, 100 × 100 = 10000 points), and if a dot is on top of the gray identity line, the power functions assume the same value for that point in the parametric space. The scenario is 2 × 2 with marginals n_1⋅ = n_2⋅ = 10 in (a) and n_1⋅ = n_2⋅ = 100 in (b).

For tables with marginal equals to (100, 100), the graphs are more concentrated near the identity line (Fig 7b), showing that all indices are more alike. The ordering still exists, but it is less severe. It is interesting to point out that, as expected, the Chi-square test works better with larger samples.

For the Hardy-Weinberg hypothesis, the results are similar to the ones obtained for the homogeneity hypothesis and are shown in Figs 8 and 9. In this case, the most powerful test was the asymptotic p-value for the LRT, followed by the exact p-value for the LRT, which is more powerful to the Chi-square test, that is similar the Barnard’s exact test. We call attention to the fact that, under hypothesis H, the power function achieves the value of 0.05, as expected, since this is the significance level chosen to build the power functions.

Fig 8. Power function for Hardy-Weinberg equilibrium hypothesis with n = 10.

Fig 9. Plots of power functions values for the Hardy-Weinberg equilibrium test.

Each graph presents one index versus another, each dot representing a point in the considered parametric space (in this case, 100 × 100 = 10000 points), and if a dot is on top of the gray identity line, the power functions assume the same value for that point in the parametric space. The scenarios are marginals n = 10 (a) and n = 100 (b).

3 Conclusion

After evaluating the indices for tables in different scenarios, we noticed that all of them had very similar behaviors, independently of the perspective (Bayesian or frequentist), sample size and table dimension. The exceptions are the p-values for Fisher and Barnard’s exact tests for the homogeneity hypothesis in 2 × 2 tables, and Barnard’s exact test for Hardy-Weinberg equilibrium, which show a discretized behavior. Studying the power functions considering homogeneity hypothesis in 2 × 2 tables and Hardy-Weinberg equilibrium hypothesis, the LRT presented itself as a powerful test when considering small sample sizes, while Fisher’s exact test was the least powerful one for the homogeneity hypothesis and the Barnard’s exact test was the least powerful for the Hardy-Weinberg equilibrium hypothesis. By enlarging sample sizes, the power of these tests increases accordingly.

Finally, we finish this paper listing our main conclusions:

• The LTR asymptotic p-value seems to be a good frequentist alternative for small sample sizes.

• Since there is an asymptotic relationship between the p-value for the LRT and the e-value (FBST), we consider that both indices are equivalent in the explored settings.

• In cases where there is available information besides the data that to be taken into account, represented by informative priors, we consider the e-value a more appropriate index than a frequenstist one, since the e-value offers a mechanism to incorporate that information.

Data Availability

The computer code that was used to generate all analysis in the paper is available from Github (https://github.com/adrianopolpo/contingencytables).

Funding Statement

This work was partially supported by the Brazilian agencies FAPESP grant 2012/16669-4, and CNPq grants 302767/2017-7 and 308776/2014-3. The agencies had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.