Exact Topological Inference for Paired Brain Networks via Persistent Homology

Abstract

We present a novel framework for characterizing paired brain networks using techniques in hyper-networks, sparse learning and persistent homology. The framework is general enough for dealing with any type of paired images such as twins, multimodal and longitudinal images. The exact nonparametric statistical inference procedure is derived on testing monotonic graph theory features that do not rely on time consuming permutation tests. The proposed method computes the exact probability in quadratic time while the permutation tests require exponential time. As illustrations, we apply the method to simulated networks and a twin fMRI study. In case of the latter, we determine the statistical significance of the heritability index of the large-scale reward network where every voxel is a network node.

1 Introduction

There are many studies related to paired images: longitudinal studies with two repeat scans [7], multimodal imaging study involving PET and MRI [13] and twin imaging studies [15]. The paired images are usually analyzed by relating voxel measurements that match across two images in a mass univariate fashion at each voxel. Compared to the paired image setting, paired brain networks have not been often analyzed possibly due to the lack of problem awareness and analysis frameworks.

In this paper, we present a new unified statistical framework that can integrate paired networks in a holistic fashion by pairing every possible combination of voxels across two images. This is achieved using a hyper-network that connects multiple smaller networks into a larger network. Although hyper-networks are frequently used in machine learning [16], the concept has not been often used in medical imaging. Jie et al. used the hyper-network framework from the resting-state fMRI in classifying MCI from AD in a machine learning framework [9]. Bezerianos et al. constructed the hyper-network from the coupling of EEG activity of pilots and copilots operating an aircraft albeit using ad-hoc procedures without an explicit model specification [1]. Motivated particularly by the science of [1], we rigorously formulate the problem of characterizing paired networks as a single hyper-network with physically nonexistent hyper-edges connecting between two existing networks.

Graph theory is often used framework for analyzing brain networks mainly due to easier accessibility and interpretation. Since we do not know the exact statistical distribution of many graph theory features, resampling techniques such as the permutation tests have been mainly used in estimating the null distribution and computing p-values. The availability of inexpensive and fast computers made the permutation tests a natural choice for computing p-values when underlying distributions are unknown. However, even with fast computers, permutation test is still extremely slow even for small-scale problems. There is a strong need for fast inference procedures. We propose a new exact inference procedure for graphs and networks. The method speeds up the computation by counting the number of permutations combinatorially instead of empirically computing them by numerically generating large number of permutations.

Our main contributions are as follows.

1. The new formulation of the hyper-network approach for paired images. We show that hyper-networks can be effectively used as a baseline model for paired twin fMRI. The proposed holistic framework is applied to a twin fMRI study in determining the statistical significance of the brain network differences and the heritability of the reward network.

2. Showing that the topological structure of the sparse hyper-network has a monotonic nestedness. This is used to define monotonic graph features and subsequently in computing topologically aware distance between networks.

3. The derivation of the probability distribution of the combinatorial test procedure on graph theory feature vectors that do not rely on resampling techniques such as the permutation tests. While the permutation tests require exponential run time, our exact combinatorial approach requires quadratic run time.

2 Sparse hyper-network

Consider a collection of n paired images

$$(x_1,y_1),(x_2,y_2),\cdots ,(x_n,y_n).$$

We assume x_k and y_k are related either genetically (twin or sibling images), scan-wise (multimodal images of the same subject) or longitudinally. We further assume (x_k, y_k) are independently but identically distributed for different i. Let x = (x₁, …, x_n)^⊤ and y = (y₁, …, y_n)^⊤ be the vectors of images. We set up a hyper-network by relating the paired vectors at different voxels υ_i and υ_j:

$$\mathbf{y}({\upsilon }_j)=\sum _{i=1}^p{\beta }_{ij}\mathbf{x}({\upsilon }_i)+\mathbf{e}$$ (1)

for some zero-mean noise vector e (Fig. 1). The parameters β = (β_ij) are the weights of the hyper-edges between voxels υ_i and υ_j that have to be estimated. We are constructing a physically nonexistent artificial network across different images. For fMRI, (1) requires estimating over billions of connections, which is computationally challenging. In practice however, each application will likely to force β to have a specific structure that may reduce the computational burden. For this study, we will consider a reduced model relevant to our genetic imaging application:

Fig. 1.

The schematic of hyper-network construction on paired image vectors x and y. The image vectors y at voxel υ_j is modeled as a linear combination of the first image vector x at all other voxels. The estimated parameters β_ij give the hyper-edge weights.

$$\mathbf{y}({\upsilon }_j)={\beta }_{ij}\mathbf{x}({\upsilon }_i)+\mathbf{e}.$$ (2)

The scientific motivation for using the reduced model will be explained in section 6. Without loss of generality, we can center and scale x and y such that

$$\sum _{k=1}^n{x}_k({\upsilon }_i)=\sum _{k=1}^n{y}_k({\upsilon }_i)=0,{\Vert \mathbf{x}({\upsilon }_i)\Vert }^2={\mathbf{x}}^{\top }({\upsilon }_i)\mathbf{x}({\upsilon }_i)={\Vert \mathbf{y}({\upsilon }_i)\Vert }^2={\mathbf{y}}^{\top }({\upsilon }_i)\mathbf{y}({\upsilon }_i)=1$$ (3)

for all voxels υ_i.

In many applications, we have significantly more number of voxels (p) than the number of images (n), so model (2) is an under-determined system and belongs to the small-n large-p problem [5]. It is necessary to regularize the model using a sparse penalty:

$$\widehat{\beta }(\lambda )=arg\underset{\beta }{min}\frac{1}{2}\sum _{i,j=1}^p{\Vert \mathbf{y}({\upsilon }_j)-{\beta }_{ij}\mathbf{x}({\upsilon }_i)\Vert }^2+\lambda \sum _{i,j=1}^p|{\beta }_{ij}|,$$ (4)

where sparse parameter λ ≥ 0 modulates the sparsity of the hyper-edges. The estimation β̂ is a function of λ. When λ = 0, (4) is a least-squares problem and we obtain β̂_ij(0) = x^⊤(υ_i)y(υ_j), which is referred to as the cross-correlation. The cross-correlation is invariant under the centering and scaling.

3 Distance between networks

To study the topological characteristic of the hyper-network, the estimated edge weights β̂_ij(λ) are binarized by assigning value 1 to any nonzero weight and 0 otherwise. Let G_λ be the resulting binary graph, i.e., unweighted graph, with adjacency matrix A_λ = (a_ij(λ)):

$$a_{ij}(\lambda )=\{\begin{array}{cc}1& \text{if}{\widehat{\beta }}_{ij}(\lambda )\ne 0;\\ 0& \text{otherwise}.\end{array}$$ (5)

As λ increases, G_λ becomes smaller and nested in a sense that every edge weight gets smaller. This is not true for the binarization of other sparse network models. Our sparse hyper-network has this extra layer of additional structure that guarantees the nestedness.

Theorem 1

The binary graph G_λ obtained from sparse model (4) satisfies

$$G_{{\lambda }_1}\supset G_{{\lambda }_2}\supset \cdots \supset G_{{\lambda }_q}$$ (6)

for any 0 ≤ λ₁ ≤ λ₂ ≤ … ≤ λ_q ≤ 1.

Proof

By solving dβ̂(λ)/dλ = 0 for λ ≠ 0 and considering β̂(0) separately it can be algebraically shown that the solution of the optimization problem (2) is given by [5]

$${\widehat{\beta }}_{ij}(\lambda )=\{\begin{array}{ccc}{\mathbf{x}}^{\top }({\upsilon }_i)\mathbf{y}({\upsilon }_j)-\lambda & \text{if}& {\mathbf{x}}^{\top }({\upsilon }_i)\mathbf{y}({\upsilon }_j)>\lambda \\ 0\hfill & \text{if}& |{\mathbf{x}}^{\top }({\upsilon }_i)\mathbf{y}({\upsilon }_j)|\le \lambda \\ {\mathbf{x}}^{\top }({\upsilon }_i)\mathbf{y}({\upsilon }_j)+\lambda & \text{if}& {\mathbf{x}}^{\top }({\upsilon }_i)\mathbf{y}({\upsilon }_j)<-\lambda \end{array}.$$ (7)

From (3) and the Cauchy-Schwarz inequality we have |x^⊤(υ_i)y(υ_j)| ≤ 1. Thus, if λ > 1, β̂(λ) = 0. To avoid the trivial case of zero edge weights everywhere, we need 0 ≤ λ ≤ 1. From (7), we have |β̂_ij(λ_j)| ≥ |β̂_ij(λ_j+1)| for 0 ≤ λ_j ≤ λ_j+1 ≤ 1. Subsequently, from (5), a_ij(λ_j) ≥ a_ij(λ_j+1). Therefore, we have G_λj ⊃ G_λj+1.

The sequence of nested multi-scale graph structure (6) is called the graph filtration in persistent homology [11]. The graph filtration can be quantified using monotonic function B satisfying

$$B(G_{{\lambda }_1})\le B(G_{{\lambda }_2})\le B(G_{{\lambda }_3})\le \cdots \le B(G_{{\lambda }_q}).$$ (8)

The number of connected components, which is the zero-th Betti number β₀ and the most often used topological invariant in persistent homology, satisfies this condition. The size of the largest connected component satisfies a similar but opposite relation of monotonic decrease.

Given two different graph filtrations ${\mathbf{G}}^1=\{G_{\lambda }^1:0\le \lambda \le 1\}$ and ${\mathbf{G}}^2=\{G_{\lambda }^2:0\le \lambda \le 1\}$, define the distance between them as

$$D({\mathbf{G}}^1,{\mathbf{G}}^2)=\underset{\lambda \in [0,1]}{sup}|B(G_{\lambda }^1)-B(G_{\lambda }^2)|,$$

which can be discretely approximated as

$$D_q=\underset{1\le j\le q}{sup}|B(G_{{\lambda }_j}^1)-B(G_{{\lambda }_j}^2)|.$$

If we choose enough number of q such that λ_j are all the sorted edge weights, then D(G¹, G²) = D_q. This is possible since there are only up to p(p – 1)/2 number of unique edge weights in a graph with p nodes and $B(G_{\lambda }^1)$ and $B(G_{\lambda }^2)$ increase discretely. In practice, ρ_j = j/q is chosen uniformly in [0, 1] or a divide-and-conquer strategy can be used to do adaptively grid the unit interval.

D satisfies all the axioms of a metric except identity. D(G¹, G²) = 0 does not imply G¹ = G². Thus, D is not a metric. However, it will be shown in section 4 that probability P(D(G¹, G²) = 0) = 0 showing such event rarely happens in practice. Thus, D can be treated as a metric-like in applications without much harm.

4 Exact topological inference

We are interested in testing the null hypothesis H₀ of the equivalence of two monotonic graph feature functions:

$$\begin{array}{c}{H}_0:B(G_{\lambda }^1)=B(G_{\lambda }^2)\text{for all}\lambda \in [0,1]\\ vs.\\ H_1:B(G_{\lambda }^1)\ne B(G_{\lambda }^2)\text{for some}\lambda \in [0,1].\end{array}$$

As a test statistic, we propose to use D_q. The test statistic takes care of the multiple comparisons by the use of supremum. Under the null hypothesis, we can derive the probability distribution of D_q combinatorially without numerically permuting images.

Theorem 2

$$P(D_q\ge d)=1-\frac{A_{q,q}}{\left(\begin{array}{c}2q\\ q\end{array}\right)},$$

where A_u,υ satisfies A_u,υ = A_u–1,υ + A_u,υ–1 with the boundary condition A_0,q = A_q,0 = 1 within band |u – υ| < d.

Proof

The proof is similar to the combinatorial construction of Kolmogorov-Smirnov (KS) test [2,5,8]. Combine two monotonically increasing vectors

$$\begin{array}{cc}(B(G_{{\rho }_1}^1),\cdots ,B(G_{{\rho }_q}^1)),& (B(G_{{\rho }_2}^2),\cdots ,B(G_{{\rho }_q}^2))\end{array}$$

and arrange them in increasing order. Represent $B(G_{{\rho }_j}^1)$ and $B(G_{{\rho }_j}^2)$ as ↑ and → respectively. For example, ↑↑→↑→→ …. There are exactly q number of ↑ and q number of → in the sequence. Treat the sequence as walks on a Cartesian grid. → indicates one step to the right and ↑ indicates one step up. Thus the walk starts at (0, 0) and ends at (q, q) (Fig. 2). There are total $\left(\begin{array}{c}2q\\ q\end{array}\right)$ possible number of paths, which forms the sample space. This is also the total number of possible permutations between the elements of the two vectors. The null assumption is that the two vectors are identical and there is no preference to one vector element to another. Thus, each walk is equally likely to happen in the sample space.

Fig. 2.

A_u,υ are computed within the boundary (dotted red line). The red numbers are the number of paths from (0, 0).

The values of B(G¹_ρj) and B(G²_ρj) satisfying the condition sup_j |B(G¹_ρj) – B(G²_ρj)| < d are the integer coordinates of (u, υ) on the path satisfying |u – υ| < d. These are integer grid points within the boundary υ = u ± d (dotted red lines in Fig. 2). Thus the probability can be written as

$$P(D_q\ge d)=1-P(D_q<d)=1-\frac{A_{q,q}}{\left(\begin{array}{c}2q\\ q\end{array}\right)},$$ (9)

where A_u,υ be the total number of passible paths from (0, 0) to (u, υ) within the boundary. Since there are only two paths (either ↑ or →), A_u,υ can be computed recursively as

$$A_{u,\upsilon }=A_{u-1,\upsilon }+A_{u,\upsilon -1}.$$

within the boundary. On the boundary A_0,q = A_q,0 = 1 since there is only one path.

Theorem 2 provides the exact probability computation for any number of nodes p. For instance, probability P(D₄ ≥ 2.5) is computed iteratively as follows. We start with computing A_1,1 = A_0,1 + A_1,0 = 2, A_2,1 = A_1,1 + A_1,0 = 3, …, A_4,4 = A_4,3 + A_3,4 = 27 + 27 = 54 (red numbers in Fig. 2). Thus the probability is $P(D_4\ge 2.5)=1-54/\left(\begin{array}{c}8\\ 4\end{array}\right)=0.23$. Few other examples that can be computed easily are

$$P(D_q=0)=0,P(D_q\ge 1)=1,P(D_q\ge q)=2/\left(\begin{array}{c}2q\\ q\end{array}\right),P(D_q\ge q+1)=0.$$

Computing A_q,q iteratively requires at most q² operations while permuting two samples consisting of q elements each requires $\left(\begin{array}{c}2q\\ q\end{array}\right)$ perations. Thus, our method can compute the p-value exactly substantially faster than the permutation test that is approximate and exponentially slow.

The asymptotic probability distribution of D_q can be also determined for sufficiently large q without computing iteratively as Theorem 2.

Theorem 3

$${lim}_{q\to \infty }P(D_q/\sqrt{2q}\ge d)=2{\sum }_{i=1}^{\infty }{(-1)}^{i-1}{e}^{-2i^2{d}^2}.$$

The proof is not given here but the result follows from [8,14]. From Theorem 3, p-value under H₀ is computed as

$$p-\text{value}=2e^{-d_o^2}-2e^{-8d_o^2}+2e^{-18d_o^2}\cdots ,$$

where d_o is the least integer greater than $D_q/\sqrt{2q}$ in the data (Fig. 3-left). For any large observed value d₀ ≥ 2, the second term is in the order of 10⁻¹⁴ and insignificant. Even for small observed d₀, the expansion converges quickly and 5 terms are sufficient.

Fig. 3.

Left: Convergence of Theorem 2 to Theorem 3 for q = 10, 50, 100, 200. Right: Run time of permutation test (dotted black line) vs. the combinational method in Theorem 2 (solid red line) in logarithmic scale.

5 Validation and comparison

The proposed method was validated and compared in two simulation studies against Gromov-Hausdorff (GH) distance. GH-distance was originally used to measure distance between two metric spaces. It was later adapted to measure distances in persistent homology, dendrograms [3] and brain networks [11]. Following [11], the computation of GH-distance is done done on graphs with1-correlation as edge weights. GH-distance is the difference in the L^∞-norm of corresponding single linkage matrices.

No network difference

The simulations were performed 1000 times and the average results were reported. There are three groups and the sample size is n = 5 in each group and the number of nodes are p = 40. In Groups I and II, data x_k(υ_i) at each node υ_i was simulated as standard normal, i.e., x_k(υ_i) ∼ N(0, 1). The paired data was simulated as y_k(υ_i) = x_k(υ_i) + N(0, 0.02²) for all the nodes (Fig. 4). Using the proposed combinatorial method, we obtained the p-values of 0.6568 ± 0.3367 and 0.2830 ± 0.3636 for β₀ and the size of the largest connected component demonstrating that the method did not detect network differences as expected.

Fig. 4.

Simulation studies. Group I and Group II are generated independently and identically. The resulting B(λ) plots are similar. The dotted red line is β₀ and the solid black line is the size of the largest connected component. No statistically significant differences can be found between Groups I and II. Group III is generated independently but identically as Group I but additional dependency is added for the first 10 nodes (square on the top left corner). The resulting B(λ) plots are statistically different between Groups I and II.

Using GH-distance, we obtained the p-value of 0.4938 ± 0.2919. The p-value was computed using the permutation test. Since there are 5 samples in each group, the total number of permutation is $\left(\begin{array}{c}10\\ 5\end{array}\right)=272$ making the permutation test exact and the comparison fair. Both methods seem to perform reasonably well. However, GH-distance method took about 950 seconds (16 minutes) while the combinatorial method took about 20 seconds in a computer.

Network difference

Group III was generated identically and independently like Group I but additional dependency was added by letting y_k(υ_j) = x_k(υ₁)/2 for ten nodes indexed by j = 1, 2, …, 10. This dependency gives high connectivity differences between Groups I and III (Fig. 4). Using the proposed combinatorial method, we obtained the p-values of 0.0422 ± 0.0908 and 0.0375 ± 0.1056 for β₀ and the size of the largest connected component. On the other hand, we obtained 0.1145 ± 0.1376 for GH-distance. The proposed method seems to perform better in the presence of signal. The MATLAB codes for performing these simulations is given in http://www.cs.wisc.edu/∼mchung/twins.

6 Twin fMRI study

The study consists of 11 monozygotic (MZ) and 9 same-sex dizygotic (DZ) twin pairs of 3T functional magnetic resonance images (fMRI) acquired in Intera-Achiava Phillips MRI scanners with a 32 channel SENSE head coil. Subjects completed monetary incentive delay task of 3 runs of 40 trials [10]. A total 120 trials consisting of 40 $0, 40 $1 and 40 $5 rewards were pseudo randomly split into 3 runs.

fMRI data went through spatial normalization to the template following the standard SPM pipeline. The fMRI dimensions after preprocessing are 53 × 63 × 46 and the voxel size is 3mm cube. There are a total of p = 25972 voxels in the template. After fitting a general linear model at each voxel, we obtained the contrast maps testing the significance of activity in the delay period for $5 trials relative to the control condition of $0 reward (Fig. 1). The paired contrast maps were then used as the initial images vectors x and y in the starting model (2).

We are interested in knowing the extent of the genetic influence on the functional brain network of these participants while anticipating the high reward as measured by activity during the delay that occurs between the reward cue and the target, and its statistical significance. The hyper-network approach is applied in extending the voxel-level univariate genetic feature called heritability index (HI) into a network-level multivariate feature. HI determines the amount of variation (in terms of percentage) due to genetic influence in a population. HI is often estimated using Falconer's formula [6] as a baseline. MZ-twins share 100% of genes while DZ-twins share 50% of genes. Thus, the additive genetic factor A, the common environmental factor C for each twin type are related as

$${\rho }_{\text{MZ}}({\upsilon }_i)=A+C,$$ (10)

$${\rho }_{\text{DZ}}({\upsilon }_i)=A/2+C,$$ (11)

where ρ_MZ and ρ_DZ are the pairwise correlation between MZ- and and same-sex DZ-twins at voxel υ_i. Solving (10) and (11), we obtain the additive genetic factor, i.e., HI given by

$$\mathcal{H}({\upsilon }_i)=2[{\rho }_{\text{MZ}}({\upsilon }_i)-{\rho }_{\text{DZ}}({\upsilon }_i)].$$

HI is a univariate feature measured at each voxel and does not quantify how the changes in one voxel is related to other voxels. We can determine HI across voxels υ_i and υ_j as

$$\mathcal{H}({\upsilon }_i,{\upsilon }_j)=2[{\varrho }_{\text{MZ}}({\upsilon }_i,{\upsilon }_j)-{\varrho }_{\text{DZ}}({\upsilon }_i,{\upsilon }_j)],$$

ϱ_MZ and ϱ_DZ are the symmetrized cross-correlations between voxels υ_i and υ_j for MZ- and DZ-twin pairs:

$$\begin{array}{cc}{\varrho }_{\text{MZ}}({\upsilon }_i,{\upsilon }_j)=({\widehat{\beta }}_{ij}^{\text{MZ}}+{\widehat{\beta }}_{ji}^{\text{MZ}})/2,& {\varrho }_{\text{DZ}}({\upsilon }_i,{\upsilon }_j)=({\widehat{\beta }}_{ij}^{\text{DZ}}+{\widehat{\beta }}_{ji}^{\text{DZ}})/2.\end{array}$$

${\widehat{\beta }}_{ij}^{\text{MZ}}$ and ${\widehat{\beta }}_{ij}^{\text{DZ}}$ are the estimated cross-correlations from model (4). Note that

$$\begin{array}{ccc}{\varrho }_{\text{MZ}}({\upsilon }_i,{\upsilon }_i)={\rho }_{\text{MZ}}({\upsilon }_i),& {\varrho }_{\text{DZ}}({\upsilon }_i,{\upsilon }_i)={\rho }_{\text{DZ}}({\upsilon }_i),& \mathcal{H}({\upsilon }_i,{\upsilon }_i)=\mathcal{H}({\upsilon }_i).\end{array}$$

In Fig. 5-left and -middle, the nodes are ρ_MZ(υ_i) and ρ_DZ(υ_i) while the edges are ϱ_MZ(υ_i, υ_j) and ϱ_DZ(υ_i, υ_j) for λ = 0.5, 0.8. In Fig. 5-right, we have ℋ(υ_i) = ℋ(υ_i, υ_i). The network visualization shows high HI values for almost everywhere along the edges. We are interested in testing the statistical significance of the estimated HI for all edges. Testing for ℋ(υ_i, υ_j) = 0 is equivalent to testing ϱ_MZ(υ_i, υ_j) = ϱ_DZ(υ_i, υ_j). Thus, we test for hyper-network differences between twins using the test statistic D_q (q = 100 is used) (Fig. 6). For β₀ and the size of the largest connected component p-values are less than 0.00002 and 0.00001 respectively indicating very strong significance of MZ- and DZ- network difference and HI of the whole brain network.

Fig. 5.

Left, middle: Node colors are correlations of MZ- and DZ-twins. Edge colors are sparse cross-correlations at sparsity λ = 0.5, 0.8. Right: Heritability index (HI) at nodes and edges. MZ-twins show higher correlations compared to DZ-twins. Some low HI nodes show high HI edges. Using only the voxel-level HI feature, we may fail to detect such high-order genetic effects on the functional network.

Fig. 6.

The result of graph filtrations on twin fMRI data. The number of connected components (left) and the size of the largest connected component (right) are plotted over the sparse parameter λ. For each λ, MZ-twins tend to have smaller number of connected components but larger connected component. The dotted green arrow (D_q) where the maximum group separation occurs.

7 Discussion

Hyper-network

In this paper, we presented a unified statistical framework for for analyzing paired images using a hyper-network. Although we applied the framework in twin fMRI, the method can be easily applied to other paired image settings such as longitudinal and multimodal studies. The method can be further extended to triple images. Then we should start with the following hyper-network model

$$\mathbf{Y}({\upsilon }_j)=\sum _{i=1}^p{\beta }_{ij}\mathbf{X}({\upsilon }_i)+\sum _{i=1}^p{\gamma }_{ij}\mathbf{Z}({\upsilon }_i)+\mathbf{e}.$$ (12)

Since we usually have more number of nodes p than the number of images n, it is necessary to introduce two separate sparse parameters in (12). Then, instead of building graph filtration over 1D, we need to build it over 2D sparse parameter space. This is related to multidimensional persistent homology [4,12]. Extending the proposed method to triple image setting is left as a future study.

Exact topological inference

We presented a combinatorial method for computing the probability that avoids time consuming permutation tests. Theorem 2 is the exact nonparametric procedure and does not assume any statistical distribution on graph features other than that they has to be monotonic. The technique is very general and applicable to any monotonic graph features such as node degree. Based on Stirling's formula, $q!~\sqrt{2\pi q}{\left(\frac{q}{e}\right)}^q$, the total number of permutations in permuting two vectors of size q each is $\left(\begin{array}{c}2q\\ q\end{array}\right)~\frac{4^q}{\sqrt{2\pi q}}$. This is substantially larger than the quadratic run time q² needed in our method (Fig. 3). Even for small q = 10, more than tens of thousands permutations are needed for the accurate estimation the p-value. On the other hand, only up to 100 iterations are needed in our combinatorial method.