Hypothesis Testing for Network Data with Power Enhancement

Abstract

Comparing two population means of network data is of paramount importance in a wide range of scientific applications. Numerous existing network inference solutions focus on global testing of entire networks, without comparing individual network links. The observed data often take the form of vectors or matrices, and the problem is formulated as comparing two covariance or precision matrices under a normal or matrix normal distribution. Moreover, many tests suffer from a limited power under a small sample size. In this article, we tackle the problem of network comparison, both global and simultaneous inferences, when the data come in a different format, i.e., in the form of a collection of symmetric matrices, each of which encodes the network structure of an individual subject. Such data format commonly arises in applications such as brain connectivity analysis and clinical genomics. We no longer require the underlying data to follow a normal distribution, but instead impose some moment conditions that are easily satisfied for numerous types of network data. Furthermore, we propose a power enhancement procedure, and show that it can control the false discovery, while it has the potential to substantially enhance the power of the test. We investigate the efficacy of our testing procedure through both an asymptotic analysis and a simulation study under a finite sample size. We further illustrate our method with examples of brain connectivity analysis.

1. Introduction

With recent prevalence of network data, the problem of comparing two populations of networks is gaining increasing attention. Our motivation is brain connectivity analysis, which studies functional and structural brain architectures through neurophysiological measures of brain activities and synchronizations (Fornito et al., 2013). Accumulated evidences have suggested that, compared to a healthy brain, the brain connectivity network alters in the presence of numerous neurological disorders, for example, Alzheimer’s disease, autism spectrum disorder, among many others. Such alternations are believed to hold crucial insights of disease pathologies (Fox and Greicius, 2010). A typical brain connectivity study collects imaging scans, such as functional magnetic resonance imaging, or diffusion tensor imaging, from groups of subjects with and without disorder. Based on the imaging scan, a network is constructed for each individual subject, with the nodes corresponding to a common set of brain regions, and the edges encoding the functional or structural associations between the regions. A fundamental scientific question of interest is to compare the brain networks and to identify local connectivity patterns that alter between the two populations. Network comparison is equally interesting in many other scientific areas as well, for instance, clinical genomics, where of crucial interest is to understand and compare gene regulatory networks of patients with and without cancer (Luscombe et al., 2004).

In the context of brain connectivity analysis, there has been a rich literature on network estimation methods (Ahn et al., 2015; Qiu et al., 2016; Wang et al., 2016; Zhu and Li, 2018, among many others). Recently, Zou et al. (2017) and Lan et al. (2018) studied estimation of the covariance matrix of a multivariate vector as a function of the similarity measure of the covariates, or a function of the adjacency matrix. There is, however, a relative paucity of inference methods, especially simultaneous inference for individual links. Even though both can produce, in effect, a concise representation of the network structure, network inference is a fundamentally different problem than network estimation. Among the few existing network inference solutions, Kim et al. (2014) studied a number of two-sample tests based on network summary metrics or generalized linear models. However, they only compared two networks globally, without any inference on the individual links of the networks. Besides, some of their tests resorted to bootstrap or permutation, which is computationally intensive and slow. Ginestet et al. (2017) characterized the geometry of the space of undirected networks with edge weights, and developed an analog of the classical two-sample test for network empirical means. However, they again focused on the global test of two entire networks. Chen et al. (2015) developed a method to detect differentially expressed connectivity subnetworks under different clinical conditions. They resorted to a permutation test, and controlled the family-wise error rate. Xia et al. (2015) first encoded the connectivity network by a partial correlation matrix computed from vector-valued data under a normal distribution. They then proposed a multiple testing procedure to compare the partial correlation matrices from the two populations, along with a proper false discovery control. Xia and Li (2019) further extended the test to matrix-valued data under a matrix normal distribution. In both cases, the test statistics were constructed based on the vector or matrix-valued data, which, as we explain next, may not be directly observable. Moreover, the underlying data distribution may not always be normal or matrix normal. Durante and Dunson (2018) developed a fully Bayesian solution for network comparison, which is very flexible and can handle the data format of our problem, but it requires specification of a series of prior distributions and can be computationally intensive.

Applications such as brain connectivity analysis actually raise new challenges for network inference. First, the observed data come in the form of p × p matrices, where p is the number of network nodes. Each such matrix encodes the network structure for one individual subject, and a collection of network samples are observed. For instance, in brain structural connectivity, what one observes are the numbers of white matter fibers between pairs of brain anatomical regions. This matrix of counts forms a network observation for one subject, with brain regions constituting the nodes and the fiber counts the links, and we observe multiple such count-valued networks for multiple subjects. This is ultimately different from the data format studied in most existing network methods, where a network structure usually takes the form a covariance or precision matrix of some vector-valued or matrix-valued data. This fundamental difference in terms of the available data format would thus require a completely new problem formulation and inferential procedure. Second, in a multitude of applications including brain connectivity analysis, the sample size is usually very small, e.g., in tens. This calls for a testing procedure that is powerful enough to detect differentially expressed links under a limited sample size. In this article, we address the problem of comparing two populations of network data, more precisely, the two population means of networks. We aim to consider both global and simultaneous inferences, tackle the new data format, and explicitly enhance the power of the test.

Specifically, suppose we observe two groups of samples, $\left\{S_{1,1},\dots ,S_{1,n_1}\right\}$ and $\left\{S_{2,1},\dots ,S_{2,n_2}\right\}$, where S_d,l denotes the observed symmetric p × p network data for the lth sample in the dth group, n_d is the total number of network samples in the dth group, l = 1, … , n_d, and d = 1, 2. Suppose $S_{d,l}={\left(S_{d,l,i,j}\right)}_{p\times p}~{\mathcal{F}}_d\left(s_d\right)$, where ${\mathcal{F}}_d$ is some distribution with a symmetric mean matrix s_d = (s_d,i,j)_p×p. Our goal is to test whether the two population means are the same:

$$H_0:s_1=s_2\text{\hspace{1em} versus \hspace{1em}}{H}_1:s_1\ne s_2.$$ (1)

If the global null in (1) is rejected, we further aim to identify at which locations the two mean matrices are different. That is, we wish to simultaneously test:

$$H_{0,i,j}:s_{1,i,j}=s_{2,i,j}\text{\hspace{1em} versus \hspace{1em}}{H}_{1,i,j}:s_{1,i,j}\ne s_{2,i,j},\text{ for }1\le i<j\le p.$$ (2)

In Xia et al. (2015), the observed data $X_{d,l}\in {ℝ}^p$ represents expressions of multiple genes for two groups of patients with long and short term survival. It is a vector, and is assumed to follow a normal distribution with the covariance matrix Σ_d. Let R_d denote the corresponding partial correlation matrix, i.e., the standardized version of ${\Sigma }_d^{-1},d=1,2$, d = 1, 2. Then the network structure is encoded by R_d, and the problem becomes testing if R₁ = R₂. Xia and Li (2019) followed a similar setup, except that the observed data $X_{d,l}\in {ℝ}^{p\times t}$ becomes a matrix, which represents brain temporal neural activity measures collected at multiple brain locations for two groups of patients with and without attention deficit hyperactivity disorder. It is assumed to follow a matrix normal distribution with the covariance Σ_d ⊗ Λ_d, and the network is still encoded by the standardized version of ${\Sigma }_d^{-1}$. The key difference for our setting is that, we do not always observe X_d,l directly, but instead S_d,l only. This difference in data format completely distinguishes our method from nearly all existing solutions such as Xia et al. (2015) and Xia and Li (2019). Moreover, we do not impose that the underlying data follows a normal or matrix normal distribution. Instead, we consider a general class of distributions for ${\mathcal{F}}_d$ satisfying some moment condition. Our method works for many different types of network links, for instance, binary links when ${\mathcal{F}}_d$ follows a light tailed distribution, or count links when ${\mathcal{F}}_d$ follows a heavy-tailed distribution.

For the global test (1), we develop a global test statistic taken as the maximum of a set of individual test statistics. We then derive its limiting null distribution, and show the resulting global test is power minimax optimal asymptotically. For the simultaneous test (2), we first develop a multiple testing procedure, and show that it can asymptotically control the false discovery at the pre-specified level. Next we propose a method to substantially enhance the power of the simultaneous inference procedure for (2). Specifically, we extend the grouping-adjusting-pooling idea of Xia et al. (2019a), and modify it for our inference of network data.

Our proposal differs from the existing solutions and makes several useful contributions. First, to the best of our knowledge, there has been no solution directly targeting simultaneous hypothesis testing of individual links for the network data in the format of S_d,l. Our method bridges this gap, and offers a timely solution to a range of scientific applications where this form of problem and data is commonly encountered. Second, our global test statistic is constructed as the maximum of the individual test statistics for all links. This type of maximum statistic enjoys various advantages and has been commonly employed in the hypothesis testing literature (e.g., Cai et al., 2013; Xia et al., 2019b). However, the derivation of its asymptotics, as well as the properties of the subsequent multiple testing procedure, are far from trivial in our new context of network comparison. Moreover, we remark that, in some network data applications, the individual test statistics may be correlated, and a global test statistic that utilizes such correlations may result in a more powerful test. However, this may not always be the case. For instance, in our brain connectivity application, the nodes are usually the brain anatomical regions, which can scatter at distant locations of the brain. As a result, there is no obvious correlation structure for the individual test statistics built on the pairs of brain regions. Therefore, we do not explicitly impose or employ any correlation structure when constructing the global test statistic. On the other hand, in our power enhancement procedure, we implicitly utilize the fact that some individual test statistics may be correlated and clustered. We then use a data driven approach to find such clusters and incorporate this information in our test. Finally, the power enhancement approach we develop is particularly useful in numerous applications, e.g., brain connectivity analysis, where the sample size is limited. Although motivated by Xia et al. (2019a), our enhancement method differs from Xia et al. (2019a) considerably in several ways. We explicitly compare the two power enhancement procedures in Section 4.5. Overall, we feel our method provides a useful addition to the general toolbox of network inference.

We adopt the following notation throughout this article. For a symmetric matrix A_d, let λ_max(A_d) and λ_min(A_d) denote the largest and smallest eigenvalues of A_d, respectively. For a set $\mathcal{H}$, let $|\mathcal{H}|$ denote its cardinality. For two sequences of real numbers {a_n} and {b_n}, write a_n = O(b_n) if there exists a constant C such that |a_n| ≤ C|b_n| holds for all n, write a_n = o(b_n) if lim_n→∞ a_n/b_n = 0, and write a_n ≍ b_n if there are positive constants c and C such that c ≤ a_n/b_n ≤ C for all n. Write n = n₁n₂/(n₁ + n₂) and assume that n₁ ≍ n₂.

The rest of the article is organized as follows. Section 2 presents the moment conditions for the distribution of ${\mathcal{F}}_d$ and show they are easily satisfied in numerous types of network data. Section 3 develops the global testing and the simultaneous testing for the two-sample network comparison, and Section 4 studies power enhancement, both of which are key to our proposal. Section 5 presents the simulations, and Section 6 presents two brain connectivity analysis examples as illustration. The Supplementary Material collects additional lemmas and the proofs.

2. Moment Conditions and Examples

We begin with some moment conditions imposed on ${\mathcal{F}}_d$. We then give a number of examples and show that those conditions are easily satisfied in numerous types of network data.

2.1. Moment conditions

We assume that the distribution ${\mathcal{F}}_d$ of the network data S_d,l satisfies one of the following two conditions: a sub-Gaussian-type tail, or a polynomial-type tail, as stated below.

(C1) (Sub-Gaussian-tail). Suppose that log p = o(n^1/5), and that there exist some constants η > 0 and K > 0, such that, for d = 1, 2,

$$\text{E}\left[\text{exp}\left\{\eta {\left(S_{d,l,i,j}-s_{d,i,j}\right)}^2/\text{Var}\left(S_{d,l,i,j}\right)\right\}\right]\le K,\text{\hspace{1em} for }1\le i<j\le p,l=1,\dots ,n_d.$$

(C2) (Polynomial-tail). Suppose that $p\le c{n}^{{\gamma }_0}$ for some constants γ₀, c > 0, and that there exist some constants ϵ > 0 and K > 0, such that, for d = 1, 2,

$$\text{E}\left\{{|\left(S_{d,l,i,j}-s_{d,i,j}\right)/\text{Var}{\left(S_{d,l,i,j}\right)}^{1/2}|}^{4{\gamma }_0+2+ϵ}\right\}\le K,\text{\hspace{1em} for }1\le i<j\le p,l=1,\dots ,n_d.$$

We first comment that, both conditions are common, and similar conditions have been often assumed in the high-dimensional setting (Cai et al., 2014; Van de Geer et al., 2014). These moment conditions are much weaker than the Gaussian assumption as usually required in the testing literature (Schott, 2007). Next we discuss a number of network examples that satisfy the above moment conditions, including Bernoulli and mixture Bernoulli data, Poisson data, correlation and partial correlation data. Furthermore, we discuss some examples where the distributions are heavy-tailed, but after some data transformation, they still satisfy the moment conditions. Examples include transformed normal count data and transformed Wishart count data.

2.2. Network data examples

The first example is binary network, which is arguably the most commonly seen network data type, where each link is a binary indicator. The Bernoulli distribution is often assumed; i.e., for S_d,l = (S_d,l,i,j)_p×p, S_d,l,i,j follows a Bernoulli distribution with mean s_d,i,j, u < s_d,i,j < 1 − u for a constant 0 < u < 1, l = 1, … , n_d, d = 1, 2, and 1 ≤ i < j ≤ p. In such case, S_d,l satisfies the sub-Gaussian-tail condition in (C1), e.g., with η = 1 and K = (1 − u) exp{u(1 − u)⁻¹} + u exp{(1−u)u⁻¹}. The same holds true for the mixture Bernoulli distribution as discussed in Durante and Dunson (2018). That is, for some integer H > 0 and randomly selected {ϕ₁, … , ϕ_H} subject to ${\sum }_{h=1}^H{\varphi }_h=1$ and ϕ_h > 0, $\text{P}\left(S_{d,l,i,j}=x\right)={\sum }_{h=1}^H{\varphi }_h{\left\{s_{d,i,j}^{(h)}\right\}}^x{\left\{1-s_{d,i,j}^{(h)}\right\}}^{1-x}$, with $u<s_{d,i,j}^{(h)}<1-u$ for some constant 0 < u < 1, x = 0, 1, h = 1, … , H, l = 1, … , n_d, d = 1, 2 and 1 ≤ i < j ≤ p. For this example, S_d,l again satisfies the sub-Gaussian-tail condition in (C1), with η = 1 and K = (1 − u) exp{u(1 − u)⁻¹} + u exp{(1 − u)u⁻¹}.

The second example is correlation network, which is another equally common network data type. In brain functional connectivity analysis and many other applications, the network is often encoded by a Pearson correlation or a partial correlation matrix. Take the Pearson correlation network as an example. The functional imaging data is usually summarized as a spatial-temporal matrix. That is, for the lth subject in the dth group, the observed data is of the form ${\mathit{X}}_{d,l}\in {ℝ}^{p\times t_d},l=1,\dots ,n_d,d=1,2$, where p is the number of brain regions, and t_d is the number of repeated measures. Then the brain functional connectivity network is encoded by the sample correlation matrix ${\mathit{S}}_{d,l}=t_d^{-1}{\sum }_{j=1}^{t_d}\left\{{\mathit{X}}_{d,l,(\cdot ,j)}-{\overline{\mathit{X}}}_{d,l}\right\}{\left\{{\mathit{X}}_{d,l,(\cdot ,j)}-{\overline{\mathit{X}}}_{d,l}\right\}}^{\top }$, where X_d,l,(·,j) denotes the jth column of the matrix X_d,l and ${\overline{\mathit{X}}}_{d,l}=t_d^{-1}{\sum }_{j=1}^{t_d}{\mathit{X}}_{d,l,(\cdot ,j)}$ denotes the sample mean vector (Fornito et al., 2013). Next we show that, as long as X_d,l satisfies one of the conditions in Lemma 1, then S_d,l satisfies the sub-Gaussian-tail condition (C1).

Lemma 1. Suppose X_d,l satisfies one of the following conditions: (i) log p = o(t^1/5), and there exist constants η′ > 0, K′ > 0 such that E(exp[η′{X_d,l,i,j − E(X_d,l,i,j)}²/Var(X_d,l,i,j)]) ≤ K′, where t = max{t₁, t₂} and t₁ ≍ t₂; (ii) $p\le c^{\prime }{t}^{{\gamma }_0^{\prime }}$, for some ${\gamma }_0^{\prime },c^{\prime }>0$ , and there exist constants ϵ′ > 0, K′ > 0 such that $E\left[{|\left\{X_{d,l,i,j}-E\left(X_{d,l,i,j}\right)\right\}/\text{Var}{\left(X_{d,l,i,j}\right)}^{1/2}|}^{4{\gamma }_0^{\prime }+4+{ϵ}^{\prime }}\right]\le K$, for i = 1, … , p, j = 1, … , t_d. Then S_d,l satisfies the sub-Gaussian-tail condition in (C1), with η = 1/4 and K = 2, as t → ∞.

We remark that a similar result as Lemma 1 can be obtained for the partial correlation network, by using the inverse regression techniques as in Liu (2013). Xia and Li (2019) tackled network comparison assuming X_d,l is directly observable and follows a matrix normal distribution. Lemma 1 suggests that, the test we develop later is still applicable when X_d,l is available, even though it may not be as powerful as the test of Xia and Li (2019) in this case. On the other hand, the main focus of this article is to develop a test of comparing two networks even when X_d,l is not observed, but only S_d,l is. As such, our test is more general than that of Xia and Li (2019).

The third example is count network, another common network data type, where each link is a count. For instance, in brain structural connectivity analysis, the link is the number of white matter fibers between anatomical brain regions. The Poisson distribution is often imposed; i.e., S_d,l,i,j follows a Poisson distribution with mean s_d,i,j, 0 < u₁ < s_d,i,j < u₂, l = 1, … , n_d, d = 1, 2, 1 ≤ i < j ≤ p. For any constant ϵ > 0, let M be the smallest integer that is no smaller than 4γ₀ + 2 + ϵ, where γ₀ is as defined in (C2). Then S_d,l satisfies the polynomial-tail condition (C2), with K upper bounded by $u_1^{-(M-1)/2}\left[{\sum }_{i=0}^M{u}_2^i\left\{\begin{array}{c}M\\ i\end{array}\right\}+u_2^M\left(u_2/2+1\right)\right]$, and $\left\{\begin{array}{c}M\\ i\end{array}\right\}$ is the number of ways to partition a set of M objects into i non-empty subsets.

We next consider some examples where the original network data ${\mathit{G}}_{d,l}={\left(G_{d,l,i,j}\right)}_{p\times p}~{\tilde{\mathcal{F}}}_d\left({\tilde{\mathit{s}}}_d\right),l=1,\dots ,n_d,d=1,2$, and ${\tilde{\mathcal{F}}}_d$ is some heavy-tailed distribution that only differs in the mean matrix ${\tilde{\mathit{s}}}_d=\left({\tilde{s}}_{d,i,j}\right)\in {ℝ}^{p\times p}$ between the two groups. In such cases, testing the means of the original samples are equivalent to testing the means of the transformed data, S_d,l,i,j = f(G_d,l,i,j), where f is some one-to-one transformation function. One example is the log-normal count network. After the logarithmic transformation of G_d,l, the transformed data S_d,l follows a normal distribution, and thus both (C1) and (C2) are satisfied. This can be further extended to the transformed normal mixture network. Another example is the transformed Wishart count network, where the transformed data S_d,l follows the Wishart distribution. For this case, S_d,l satisfies the sub-Gaussian-tail condition (C1). Moreover, in this case, the testing problems (1) and (2) are closely related to the covariance matrix testing problems studied in Li and Chen (2012) and Cai et al. (2013). The key difference between our method and the existing ones is that, we only observe S_d,l, but not the original vector samples. This example can be further extended to the case of the product of Gaussian mixtures network, or the Wishart mixtures network.

3. Two-sample Test on Network Data

We begin with the construction of a test statistic for the two testing problems (1) and (2). We then develop a global testing procedure for (1), and a simultaneous testing procedure for (2). For each test, we derive its corresponding asymptotic properties.

3.1. Test statistics

We first observe that the testing problem (1) is equivalent to the test, $H_0^{\prime }:{\text{max}}_{1\le i<j\le p}|s_{1,i,j}-s_{2,i,j}|=0$. This motivates us to construct the test statistic based on

$$W_{i,j}={\overline{S}}_{1,i,j}-{\overline{S}}_{2,i,j}.$$

where ${\overline{S}}_{d,i,j}=n_d^{-1}{\sum }_{l=1}^{n_d}{S}_{d,l,i,j}$. We standardize W_i,j, and estimate the variance of S_d,l,i,j by

$$V_{1,i,j}=n_1^{-1}\sum _{l=1}^{n_1}{\left(S_{1,l,i,j}-{\overline{S}}_{1,i,j}\right)}^2,\text{ and }{V}_{2,i,j}=n_2^{-1}\sum _{l=1}^{n_2}{\left(S_{2,l,i,j}-{\overline{S}}_{2,i,j}\right)}^2,$$ (3)

respectively. This leads to our test statistic,

$$T_{i,j}=\frac{W_{i,j}}{{\left(V_{1,i,j}/n_1+V_{2,i,j}/n_2\right)}^{1/2}},1\le i<j\le p.$$ (4)

3.2. Global test

In brain connectivity analysis and many other applications, it is generally postulated that the differences between two network structures concentrate on a small number of brain regions. This translates to a sparse alternative in our global test. Correspondingly, we construct the global test statistic as,

$$M_n=\underset{1\le i<j\le p}{\text{max}}{T}_{i,j}^2.$$

Let ${\Gamma }_d\in {ℝ}^{q\times q}$ denote the covariance matrix of vech(S_d,l), where q = p(p − 1)/2, and vech(·) is the operator that turns the upper triangular part of S_d,l into a vector. Let ${\mathit{R}}_d=\left(r_{d,i,j}\right)\in {ℝ}^{q\times q}$ denote the corresponding correlation matrix. We introduce two conditions.

(A1) $C_0^{-1}\le {\lambda }_{\text{min}}\left({\Gamma }_d\right)\le {\lambda }_{\text{max}}\left({\Gamma }_d\right)\le C_0$ for some constant C₀ > 0, d = 1, 2.

(A2) max_d=1,2 max_1≤i<j≤q |r_d,i,j| < r < 1 for some constant 0 < r < 1.

Both conditions are mild. Particularly, Condition (A1) implies that ${\text{max}}_j{s}_j\left({\alpha }_0\right)\le K{c}_q^{-2}$ for some constant K > 0, where s_j(α₀) = |{i : max_d=1,2 |r_d,i,j| ≥ c_q}| and c_q is a correlation order that depends on q, with a common choice of ${(\text{log }q)}^{-1-{\alpha }_0}$ for some α₀ > 0. In other words, it allows at most $O\left\{q{c}_q^{-2}\right\}$ highly correlated pairs of network entries. For the high-dimensional vector-valued data, such a condition on the eigenvalues of the covariance matrix is commonly imposed (Bickel et al., 2008; Rothman et al., 2008; Yuan, 2010; Cai et al., 2014). Condition (A2) is also mild, because if max_1≤i<j≤q |r_d,i,j| = 1, then Γ_d is singular. We next obtain the limiting distribution of our test statistic M_n.

Theorem 1. Suppose that (A1)-(A2), and one of (C1) and (C2) hold. Then for any $x\in ℝ$,

$$P_{H_0}\left(M_n-2\text{\hspace{0.17em}log\hspace{0.17em}}q+\text{log\hspace{0.17em}log\hspace{0.17em}}q\le x\right)\to \text{exp}\left\{-{\pi }^{-1/2}\text{\hspace{0.17em}exp}(-x/2)\right\},\mathit{\text{ as }}{n}_1,n_2,q\to \infty .$$

Based on this limiting null distribution, we define the asymptotic α-level test as,

$${\Psi }_{\alpha }=I\left(M_n\ge 2\text{\hspace{0.17em}log\hspace{0.17em}}q-\text{log\hspace{0.17em}log\hspace{0.17em}}q+q_{\alpha }\right),$$

where q_α = −log π − 2 log log(1 − α)⁻¹.

We next study the power and the asymptotic optimality of the test Ψ_α. Toward that end, define the sparsity of s₁ − s₂ as k_q = |{(i, j) : s_1,i,j − s_2,i,j ≠ 0, 1 ≤ i < j ≤ p}|. We also introduce a class of (s₁, s₂),

$$\mathcal{U}(c)=\left\{\left({\mathit{s}}_1,{\mathit{s}}_2\right):\underset{1\le i<j\le p}{\text{max}}\frac{|s_{1,i,j}-s_{2,i,j}|}{{\left\{\text{Var}\left(S_{1,l,i,j}\right)/n_1+\text{Var}\left(S_{2,l,i,j}\right)/n_2\right\}}^{1/2}}\ge c{(\text{log\hspace{0.17em}}q)}^{1/2}\right\}.$$

Theorem 2. Suppose that one of (C1) and (C2) holds. Then,

$$\underset{\left(s_1,s_2\right)\in \mathcal{U}(2\sqrt{2})}{\text{inf}}\mathit{P}\left({\Psi }_{\alpha }=1\right)\to 1,\text{ as }{n}_1,n_2,q\to \infty .$$

Furthermore, suppose that k_q = o(q^r) for some r < 1/2. Let α, β > 0 and α + β = 1. Then there exists a constant c₀ > 0 such that, for all sufficiently large n_d and q,

$$\underset{\left(s_1,s_2\right)\in U\left(c_0\right)}{\text{inf}}\underset{T_{\alpha }\in {\mathcal{T}}_{\alpha }}{\text{sup}}\mathit{P}\left(T_{\alpha }=1\right)\le 1-\beta ,$$

where ${\mathcal{T}}_{\alpha }$ is the set of all α-level tests, i.e.,$P_{H_0}\left(T_{\alpha }=1\right)\le \alpha$ for all $T_{\alpha }\in {\mathcal{T}}_{\alpha }$.

This theorem shows that the null hypothesis in (1) can be rejected by Ψ_α with a high probability if the pair of the network means belong to the class $\mathcal{U}(2\sqrt{2})$. In addition, with the mild sparsity condition k_q = o(q^r), the lower bound rate of (log q)^1/2 cannot be further improved, because for a sufficiently small c₀, any α-level test is unable to reject the null correctly uniformly over $\mathcal{U}\left(c_0\right)$ with probability tending to 1. Henceforth, the global test Ψ_α reaches the power minimax optimality asymptotically.

3.3. Simultaneous test

We next develop a multiple testing procedure for (2) based on the test statistic T_i,j in (4). Let h be the threshold level such that H_0,i,j is rejected if |T_i,j| ≥ h. Let ${\mathcal{H}}_0=\left\{\left(i,j\right):s_{1,i,j}=s_{2,i,j},1\le i<j\le p\right\}$ be the set of true nulls, and ${\mathcal{H}}_1=\mathcal{H}\backslash {\mathcal{H}}_0$ the set of true alternatives, where $\mathcal{H}=\{(i,j):1\le i<j\le p\}$. Denote by $R_0(h)={\sum }_{(i,j)\in {\mathcal{H}}_0}I\left(|T_{i,j}|\ge h\right)$ and $R(h)={\sum }_{1\le i<j\le p}I\left(|T_{i,j}|\ge h\right)$ the total number of false positives and rejections, respectively. Then we define the false discovery proportion and false discovery rate by

$$\text{FDP}(h)=\frac{R_0(h)}{R(h)\vee 1},\text{\hspace{1em}FDR}(h)=\text{E}\left\{\text{FDP}(h)\right\}.$$

An ideal choice of h would reject as many true positives as possible while controlling the FDP at the pre-specified level α. That is, we select h₀ = inf {h : 0 ≤ h ≤ (2 log q)^1/2, FDP(h) ≤ α}. Since R₀(h) is unknown, we estimate it conservatively by 2q{1 − Φ(h)}, where Φ(h) is the standard normal cumulative distribution function. This leads to our multiple testing procedure as summarized in Algorithm 1.

We next show that this testing procedure controls the FDR and FDP asymptotically at the pre-specified level. For notation simplicity, we write $\text{FDP}=\text{FDP}(\widehat{h})$ and $\text{FDR}=\text{FDR}(\widehat{h})$, where $\widehat{h}$ is obtained in Algorithm 1. Define ${\mathcal{A}}_i(\xi )=\left\{j:\text{max}\left(|r_{1,i,j}|,|r_{2,i,j}|\right)\ge {(\text{log\hspace{0.17em}}q)}^{-2-\xi }\right\}$, and ${\mathcal{S}}_{\rho }=\left\{(i,j):1\le i<j\le p,|s_{1,i,j}-s_{2,i,j}|/{\left\{\text{Var}\left(S_{1,l,i,j}\right)/n_1+\text{Var}\left(S_{2,l,i,j}\right)/n_2\right\}}^{1/2}\ge {(\text{log }q)}^{1/2+\rho }\right\}$. We further introduce some conditions.

Algorithm 1.

Simultaneous inference with FDR control

Step 1: Estimate FDP by $\widehat{\text{FDP}}(h)=2q\left\{1-\Phi (h)\right\}/\left\{R(h)\vee 1\right\}$. Step 2: For a given 0 ≤ α ≤ 1, calculate $$\widehat{h}=\text{inf}\left\{h:0\le h\le {(2\text{\hspace{0.17em}log\hspace{0.17em}}q)}^{1/2},\widehat{\text{FDP}}(h)\le \alpha \right\}.$$ If $\widehat{h}$ does not exist, set $\widehat{h}={(2\text{\hspace{0.17em}log\hspace{0.17em}}q)}^{1/2}$. Step 3: Reject H0,i,j if and only if $|T_{i,j}|\ge \widehat{h}$, for 1 ≤ i ≤ j ≤ p.

(B1) $|{\mathcal{S}}_{\rho }|\ge \left[1/\left\{{\pi }^{1/2}\alpha \right\}+\delta \right]{(\text{log }q)}^{1/2}$, for some constant δ > 0 and any sufficiently small constant ρ > 0.

(B2) ${\text{max}}_{1\le i\le q}|{\mathcal{A}}_i(\xi )|=o\left(q^{\nu }\right)$ for some constants ξ > 0 and 0 < ν < (1 − r)/(1 + r).

(B3) $q_0=|{\mathcal{H}}_0|\ge c_{1}q$ for some constant c₁ > 0.

Condition (B1) on ${\mathcal{S}}_{\rho }$ is mild, as it only requires a small number of s₁ and s₂ having standardized difference with the order of (log q)^1/2+ρ for any sufficiently small constant ρ > 0. Condition (B2) is mild, as it requires that not too many S_d,l,i,j are highly correlated, but still allows the number of highly correlated pairs to grow in the order of o(q^1+ν). Condition (B3) is also a natural and mild assumption, because if it does not hold, i.e., q₀ = o(q), then we can simply reject all the hypotheses. As a result, we would have |R₀| = q₀, |R| = q, and the FDR would tend to zero. Under these conditions, we obtain the asymptotic properties of our multiple testing procedure in terms of false discovery control.

Theorem 3. Suppose that (A2), (B1)-(B3), and one of (C1) and (C2) hold with $p\le c{n}^{{\gamma }_0}$ for some constants γ₀, c > 0. Then,

$$\underset{\left(n_1,n_2,q\right)\to \infty }{\text{lim}}\frac{\mathit{\text{FDR}}}{\alpha q_0/q}=1,\mathit{\text{ and \hspace{1em}}}\frac{\mathit{\text{FDP}}}{\alpha q_0/q}\to 1\mathit{\text{ in probability, as }}{n}_1,n_2,q\to \infty .$$

4. Power Enhancement

In brain connectivity analysis and many other applications, the sample size n_d is often small, whereas the number of nodes p can be moderate to large. This results in a limited power for the proposed test. We explore in this section an explicit power enhancement method that has potential to substantially improve the power of the simultaneous inference developed in Section 3.3. We borrow the idea of grouping, adjusting and pooling (GAP) that was first proposed in Xia et al. (2019a). However, our method differs from Xia et al. (2019a) in many ways, including a different, and actually less restrictive, assumption, a different set of primary and auxiliary statistics, and a different modification of the multiple testing procedure. We show that the modified procedure is asymptotically more powerful, meanwhile it can still control FDR and FDP asymptotically. We obtain these properties assuming the sub-Gaussian-tail condition (C1). Parallel results can be obtained under the polynomial-tail condition (C2) too, but are technically more involved. We begin by describing the intuition behind our power enhancement solution, then derive the proper auxiliary statistic for our inference problem. We then develop the modified simultaneous testing procedure, and study its asymptotic properties in terms of power improvement and false discovery control. We also compare in detail our method with the GAP method of Xia et al. (2019a).

4.1. Intuition

We recognize that there exists additional information in the data that is potentially useful to improve the simultaneous testing procedure of Algorithm 1. We first discuss our intuition, then use some simple example to illustrate where the auxiliary information is and how it can facilitate our multiple testing procedure.

In a multitude of applications including brain connectivity analysis, it is often believed that the difference between the two networks under different biological conditions is small. This means s₁ − s₂ is sparse. Accordingly, one can find a baseline matrix s₀, such that ${\mathit{s}}_1^{\prime }={\mathit{s}}_1-{\mathit{s}}_0$ and ${\mathit{s}}_2^{\prime }={\mathit{s}}_2-{\mathit{s}}_0$ are individually sparse. Let ${\mathcal{I}}_d=\left\{(i,j):s_{d,i,j}^{\prime }\ne 0,1\le i<j\le p\right\}$ denote the support of ${\mathit{s}}_d^{\prime }$, d = 1, 2, and $\mathcal{I}={\mathcal{I}}_1\cup {\mathcal{I}}_2$ denote the union support. Note that the set of alternative hypotheses ${\mathcal{H}}_1$ defined in Section 3.3 is the same as $\mathcal{I}$, if s_1,i,j ≠ s_2,i,j for every $(i,j)\in {\mathcal{I}}_1\cap {\mathcal{I}}_2$. In general, ${\mathcal{H}}_1$ is a proper subset of $\mathcal{I}$. Since ${\mathit{s}}_1^{\prime }$ and ${\mathit{s}}_2^{\prime }$ are both sparse, we realize that the cardinality of $\mathcal{I}$ is small. Moreover, the following relationship holds true:

$$(i,j)\notin \mathcal{I}\text{\hspace{1em} implies that }{s}_{1,i,j}-s_{2,i,j}=0,1\le i<j\le p.$$

Therefore, the knowledge about $\mathcal{I}$ is useful to help narrow down the search in multiple testing. In other words, if one can find a way to identify possible entries (i, j) in $\mathcal{I}$, it would provide useful information about the set of true alternatives ${\mathcal{H}}_1$, or equivalently, the set of true nulls ${\mathcal{H}}_0$. As a consequence, it can potentially increase the power of the testing procedure.

A key observation is then, while the test statistic is built on the difference between ${\overline{S}}_{1,i,j}$ and ${\overline{S}}_{2,i,j}$ as defined in Section 3.1, the sum of ${\overline{S}}_{1,i,j}$ and ${\overline{S}}_{2,i,j}$ can provide crucial information about $\mathcal{I}$. Consider a toy example where the network data is binary, and S_d,l,i,j follows a Bernoulli distribution with mean s_d,l,i,j, l = 1, … , n_d, d = 1, 2, 1 ≤ i < j ≤ p. Assume that s_1,i,j = s_2,i,j = s_0,i,j = 0.1 for 80% of the (i, j) pairs, s_1,i,j = s_2,i,j = s_0,i,j = 0.9 for 10% of the (i, j) pairs, and for the rest of the (i, j) pairs, s_1,i,j, s_2,i,j ~ Uniform(0.1, 0.9) and s_0,i,j = 0.1. In this example, for the pairs $(i,j)\notin \mathcal{I}$, the sum of s_1,i,j and s_2,i,j is either very small, which is 0.2, or very large, i.e., 1.8. Meanwhile, for the pairs $(i,j)\notin \mathcal{I}$, the sum is in between. Henceforth, this sum contains useful information about $\mathcal{I}$, and can potentially enhance the power of the multiple testing procedure.

Based on the above discussion, we can see that, the more sparsity structure information the auxiliary statistics can capture, the more information they can provide about the union support $\mathcal{I}$, and the more substantial power gain the test can achieve. In general, the sparser the true difference s₁ − s₂ is, the more information the auxiliary statistics can offer.

4.2. Auxiliary statistics

We next formally construct the auxiliary statistic that provides useful information about the union support $\mathcal{I}$. It is important to note that, the auxiliary statistic should be constructed so that they are asymptotically independent of the test statistic T_i,j in (4). This way the null distribution of T_i,j would not be distorted by the incorporation of the auxiliary statistic.

Recall V_d,i,j in (3) is the sample variance of S_d,l,i,j. We construct the auxiliary statistic as,

$$A_{i,j}=\frac{{\overline{S}}_{1,i,j}+{\widehat{\kappa }}_{i,j}{\overline{S}}_{2,i,j}}{{\left(V_{1,i,j}/n_1+{\widehat{\kappa }}_{i,j}^2{V}_{2,i,j}/n_2\right)}^{1/2}},1\le i<j\le p,$$

where ${\widehat{\kappa }}_{i,j}=\left(n_2{V}_{1,i,j}\right)/\left(n_1{V}_{2,i,j}\right)$. The next proposition shows that the test statistic T_i,j and the auxiliary statistic A_i,j are asymptotically independent under the null hypothesis. Define

$$a_{i,j}=\frac{s_{1,i,j}+{\kappa }_{i,j}{s}_{2,i,j}}{{\left\{\text{Var}\left(S_{1,l,i,j}\right)+{\kappa }_{i,j}^2\text{\hspace{0.17em}Var}\left(S_{1,l,i,j}\right)\right\}}^{1/2}},\text{\hspace{1em} where }{\kappa }_{i,j}=\frac{n_2\text{Var}\left(S_{1,l,i,j}\right)}{n_1\text{Var}\left(S_{2,l,i,j}\right)}.$$

Proposition 1. Suppose (C1) holds with log q = o(n^1/c) for some c > 5. For any constants M > 0 and C > 0, we have

$$P_{H_{0,i,j}}\left(|T_{i,j}|\ge h,|A_{i,j}|\ge \lambda \right)=\left\{1+o(1)\right\}G(h)P\left(|N(0,1)+a_{i,j}|\ge \lambda \right)+O\left(q^{-M}\right),$$

uniformly for $0\le h\le C\sqrt{\text{log }q}$, $0\le \lambda \le C\sqrt{\text{log }q}$, and 1 ≤ i < j ≤ p, with G(h) = 2{1 − Φ(h)}. Furthermore, for all 0 ≤ k ≤ CN with an integer constant N,

$$P_{H_{0,i,j}}\left(|T_{i,j}|\ge h,|A_{i,j}|<{\lambda }_k\right)=\left\{1+o(1)\right\}G(h)P\left(|N(0,1)+a_{i,j}|<{\lambda }_k\right)+O\left(q^{-M}\right),$$

uniformly for $0\le h\le C\sqrt{\text{log}q}$ and 1 ≤ i < j ≤ p, where ${\lambda }_k=(k/N)\sqrt{\text{log }q}$.

4.3. Power enhanced simultaneous test

Based on (T_i,j, A_i,j), we now modify the simultaneous testing procedure of Algorithm 1. We first describe the main idea. We next summarize the modified testing procedure in Algorithm 2. Finally, we discuss some specific choices of the key parameters of the algorithm.

Since there are totally q = p(p − 1)/2 tests to carry out simultaneously, we rearrange the pairs of {(T_i,j, A_i,j), 1 ≤ i < j ≤ p} into {(T_i, A_i), i = 1, … , q}. After obtaining all the p-values, p_i = 2{1 − Φ(|T_i|)}, from Algorithm 1, our basic idea is to adjust those p-values by $p_i^w=\text{min}\left\{p_i/w_i,1\right\}$, with w_i being the adjusting weights, i = 1, … , q. We utilize the auxiliary statistics A_i to help compute the adjusting weights w_i, by groups. Specifically, we consider a set of grid values, $\mathcal{J}=\left\{\left(C_{1}N-1\right)\sqrt{\text{log }q}/N,C_1\sqrt{\text{log }q},\dots ,\left(C_{2}N-1\right)\sqrt{\text{log }q}/N,C_2\sqrt{\text{log }q}\right\}$, where C₁, C₂ and N are some pre-specified constants. We divide the index set {1, … , q} into K groups according to the auxiliary statistics (A₁, … , A_q). As an example, we take K = 3. That is, we choose two grid points ${\mathcal{J}}_K=\left\{{\lambda }_1,{\lambda }_2\right\}$ in $\mathcal{J}$, and obtain K = 3 groups of indices, ${\mathcal{G}}_1=\left\{i:1\le i\le q,-\infty <A_i\le {\lambda }_1\right\}$, ${\mathcal{G}}_2=\left\{i:1\le i\le q,{\lambda }_1<A_i\le {\lambda }_2\right\}$, and ${\mathcal{G}}_3=\left\{i:1\le i\le q,{\lambda }_2<A_i\le \infty \right\}$. For each group ${\mathcal{G}}_k$, we compute its cardinality, $q_k=|{\mathcal{G}}_k|$. We also estimate the proportion, π_k, of alternatives in ${\mathcal{G}}_k$, k = 1, … , K. To do so, we employ the method of Schweder and Spjøtvoll (1982) and Storey (2002) to obtain an estimate ${\tilde{\pi }}_k$ first, then stablize it by ${\widehat{\pi }}_k=\left(ϵ\vee {\tilde{\pi }}_k\right)\wedge (1-ϵ)$, where ϵ is a small positive number; we set ϵ = 10⁻⁵. Then for all the indices in ${\mathcal{G}}_k$, we compute the group-wise adjusting weight:

$$w_i={\left(\sum _{k=1}^K\frac{q_k{\widehat{\pi }}_k}{1-{\widehat{\pi }}_k}\right)}^{-1}\frac{q{\widehat{\pi }}_k}{\left(1-{\widehat{\pi }}_k\right)},i\in {\mathcal{G}}_k,1\le k\le K.$$ (5)

This idea of adjusting the weights w_i by groups is motivated by our intuition in Section 4.1. After obtaining the weights, we adjust the p-values and apply the Benjamini-Hochberg procedure (Benjamini and Hochberg, 1995, BH) to the adjusted p-values $p_i^w$. Finally, we search all possible choices of ${\mathcal{J}}_K$ among $\mathcal{J}$, and find the one that yields the largest number of rejections. We apply BH again to the adjusted p-values under this choice of ${\mathcal{J}}_K$ to obtain the final adjusted rejection region. We summarize this modified simultaneous testing procedure in Algorithm 2.

We discuss some specific choices of the parameters in Algorithm 2. First, the number of groups K is usually set at K = 3. As shown in Xia et al. (2019a), when K ≤ 4, there is little additional power gain, but a more expensive computation. Second, the constants C₁ and C₂ can be chosen so that $C_1\sqrt{\text{log }q}$ is equal to the smallest value of the auxiliary statistics and is equal to the largest value of the auxiliary statistics. If the absolute values of the smallest and largest auxiliary statistics exceed $C_2\sqrt{\text{log }q}$, we truncate at C₁ = −16 and C₂ = 16 to stabilize and expedite the computation. We note here that, if the network data are non-negative, such as the binary and poisson network data, then both C₁ and C₂ are non-negative. By contrast, in Xia et al. (2019a), C₁ and C₂ were fixed at −4 and 4. Finally, N can be any integer for the theoretical validity. Numerically, a larger value of N implies a more precise grid search, but at the cost of a heavier computational burden. We choose N such that the gap between two adjacent grid points, (log p)^1/2/N, equals 0.1 approximately.

Algorithm 2.

Adjusted simultaneous inference with FDR control and power enhancement.

Step 1: Initialization: Step 1.1: Compute the test statistics and auxiliary statistics {(Ti,, Ai); i = 1,…, q}. Step 1.2: Compute thep-values: pi = 2{1 − Φ(|Ti|)}, i,…,.q. Step 1.3: Input the pre-specified constants K, C1, C2 and N. Step 1.4: Compute the grid set: $$\mathcal{J}=\left\{\left(C_{1}N-1\right)\sqrt{\text{log\hspace{0.17em}}q}/N,C_1\sqrt{\text{log }q},\dots ,\left(C_{2}N-1\right)\sqrt{\text{log }q}/N,C_2\sqrt{\text{log }q}\right\}.$$ Step 2: For each ${\mathcal{J}}_K=\left\{{\lambda }_1,\dots ,{\lambda }_{K-1}\right\}$ in $\mathcal{J}$ and λo = −∞, λK = ∞: Step 2.1: Construct ${\mathcal{G}}_k=\left\{i:1\le i\le q,{\lambda }_{k-1}<A_i\le {\lambda }_k\right\},1\le k\le K$. Step 2.2: For each ${\mathcal{G}}_k$, compute the cardinality, $q_k=|{\mathcal{G}}_k|$. Step 2:3: For each ${\mathcal{G}}_k$, estimate the proportion, ${\widehat{\pi }}_k$, of alternatives in ${\mathcal{G}}_k$. Step 2.4: Compute the adjusting weights Wi, i = 1,…, q, according to (5). Step 2.5: Adjust the p-values: $p_i^w=\text{min}\left\{p_i/w_i,1\right\},i=1,\dots ,q$. Step 2.6: Apply the BH procedure, and record the total number of rejections. Step 3: Obtain the adjusted rejection region: Step 3.1: Choose ${\mathcal{J}}_K$ that yields the largest number of rejections. Step 3.2: Compute the corresponding adjusted p-values: $p_i^w,1\le i\le q$. Step 3.3: Reorder all the adjusted p- values: $p_{(1)}^w\le \dots \le p_{(q)}^w$. Step 3.4: Output the rejection region $\{i:i<\widehat{\tau }\}$, where $\widehat{\tau }=\text{max}\left\{i:p_{(i)}^w\le \alpha i/q\right\}$.

4.4. FDR control and power enhancement

We next show that the modified inference of Algorithm 2 is asymptotically more powerful than Algorithm 1, meanwhile it can still asymptotically control the false discovery.

Denote $\left\{p_i^w:1\le i\le q\right\}$ the adjusted p-values from Algorithm 2, and $\left\{p_{(i)}^w:1\le i\le q\right\}$ the ordered adjusted p-values. The corresponding adjusted FDP is:

$$\mathrm{F}\mathrm{D}{\mathrm{P}}_{\text{adj}}=\frac{{\sum }_{i\in {\mathcal{H}}_0}I\left\{p_i^w\le p_{(\widehat{\tau })}^w\right\}}{{\sum }_{i=1}^{q}I\left\{p_i^w\le p_{(\widehat{\tau })}^w\right\}\vee 1},$$

where $\widehat{\mathcal{T}}$ is the cutoff obtained from Step 3.4 of Algorithm 2, and I(·) is the indicator function. Accordingly, FDR_adj = E(FDP_adj). The next theorem shows that the modified procedure can still control FDR and FDP asymptotically.

Theorem 4. Suppose (A2), (B1)-(B3), and (C1) hold with $p\le c{n}^{{\gamma }_0}$ for constants γ₀, c > 0. Then,

$$\underset{\left(n_1,n_2,q\right)\to \infty }{\text{lim}}\frac{{\mathit{\text{FDR}}}_{adj}}{\alpha q_0/q}=1,\mathit{\text{ and \hspace{1em}}}\frac{{\mathit{\text{FDP}}}_{adj}}{\alpha q_0/q}\to 1\mathit{\text{ in probability, as }}{n}_1,n_2,q\to \infty .$$

Next, denote the power of the testing procedures of Algorithms 1 and 2 by Ψ and Ψ_adj, respectively. That is,

$$\Psi =\text{E}\left\{\frac{\sum _{(i,j)\in {\mathcal{H}}_1}I\left(|T_{i,j}|\ge \widehat{h}\right)}{|{\mathcal{H}}_1|}\right\},{\Psi }_{\text{adj}}=\text{E}\left[\frac{\sum _{i\in {\mathcal{H}}_1}I\left\{p_i^w\le p_{(\widehat{\tau })}^w\right\}}{|{\mathcal{H}}_1|}\right].$$

Then the next theorem shows that, by incorporating the auxiliary statistics A_i,j, the modified simultaneous testing procedure of Algorithm 2 is asymptotically more powerful than Algorithm 1, which is solely based on the test statistics T_i,j.

Theorem 5. Suppose the same conditions in Theorem 4 hold. Then,

$${\Psi }_{\text{adj}}\ge \Psi +o(1),\mathit{\text{ as \hspace{1em}}}q\to \infty .$$

4.5. Comparison to GAP

Although motivated by the GAP method of Xia et al. (2019a), our power enhancement procedure is also considerably different. While GAP tackled the problem of mean comparison of vector-valued samples, we target the problem of network mean comparison. This leads to a different set of test and auxiliary statistics, but a number of additional intrinsic differences as well.

First, the two methods impose different assumptions. A key requirement for GAP to enhance the power is that the parameters of interest from each group are individually sparse. In our setup, however, the parameters may all be non-negative. For instance, in a binary network or a count network, all the entries of both means s₁ and s₂ are usually non-negative. As such, the means may not be individually sparse. Our procedure instead only requires the difference of the two means s₁ − s₂ is sparse, which reasonably holds and is often imposed in numerous applications including brain connectivity analysis (Zhu and Li, 2018).

Second, the two methods differ in terms of the range of the auxiliary statistics that contribute most to the power enhancement. Consider the case when K = 3. In Xia et al. (2019a), since both means are assumed to be individually sparse, the tests that are more likely to be adjusted and rejected are those with the corresponding auxiliary statistics either being negative and small, or positive and large. That is, the power enhancement hinges more on those tests in ${\mathcal{G}}_1$ and ${\mathcal{G}}_3$ with small or large auxiliary statistics. However, in our setup, the individual means s₁ and s₂ can both be dense and their entries are all positive. Instead we only assume that s₁ − s₂ is sparse. Take a binary brain connectivity network as an example. The observed networks are often sparse, in that most links are zero, since it is known that brain connections are energy consuming and biological units tend to minimize energy-consuming activities (Raichle and Gusnard, 2002; Bullmore and Sporns, 2009). This translates to small connection probabilities for most entries of s₁ and s₂, while all these probabilities are positive. Moreover, the difference of the means between the two populations is often sparse, which translates to equal connection probabilities for most entries of s₁ and s₂, or equivalently zero difference for most entries of of s₁ − s₂. This is similar to the toy example we discuss in Section 4.1. For such cases, as a consequence of Algorithm 2, the tests whose corresponding auxiliary statistics are too small or too large would be adjusted so that they are less likely to be rejected. Instead, those tests whose auxiliary statistics are in between would be adjusted so that they are more likely to be rejected. In other words, the power enhancement in our setup may hinge more on ${\mathcal{G}}_2$, rather than ${\mathcal{G}}_1$ and ${\mathcal{G}}_3$.

Third, due to the above difference, the grid construction in Step 1.4 of Algorithm 2 is noticeably different from that of GAP in Xia et al. (2019a). Specifically, in Xia et al. (2019a), to ensure the inclusion of important locations in ${\mathcal{G}}_1$ and ${\mathcal{G}}_3$, the constants C₁ and C₂ can be simply fixed at −4 and 4, respectively, so that the upper bound of those small negative auxiliary statistics and lower bound of those large positive auxiliary statistics can be attained in the grid $\mathcal{J}$. By contrast, for our problem, the upper bound of the auxiliary statistics in the union support $\mathcal{I}$ can go beyond the bound in Xia et al. (2019a), i.e., $4\sqrt{\text{log }q}$, and the lower bound of the auxiliary statistics in the union support can be non-negative. Since the tests in ${\mathcal{G}}_2$ are more likely to be adjusted and rejected, we need to do a more thorough grid construction and choose the constants C₁ and C₂ based on the smallest and largest values of the auxiliary statistics as described in Section 4.3.

5. Simulations

We first present the simulation setup, where we consider different network structures, sparsity levels, network sizes and sample sizes. We then investigate the empirical performance of the global test, and compare the two simultaneous tests, Algorithms 1 and 2.

5.1. Setup

We consider p × p networks, with two network sizes, p = 100 and 200. This results in q = 100(100 − 1)/2 = 4590 and q = 200(200 − 1)/2 = 19900 links, respectively. We consider five common network structures, the Bernoulli, Bernoulli mixture, and transformed Wishart distributions, and for the Bernoulli case, the binary links are generated from a power-law distribution, a stochastic block model, and an Erdös-Rényi model. For each network structure, we further consider three sparsity levels.

• + Bernoulli: Select the sets ${\mathcal{M}}_{d,1}$ and ${\mathcal{M}}_0$ from q hypotheses according to the following models generated by the R package igraph, with $|{\mathcal{M}}_{d,1}|=|{\mathcal{M}}_0|=k_q/2$, d = 1, 2. Here k_q is a parameter that controls the sparsity level, and is specified later.

  • − Power-law distribution: with p nodes, k_q/2 edges, the power law exponent of the degree distribution is set to 2.1, and all other parameters are set to the default values.
  • − Stochastic block model: with 2 blocks and the diagonal Bernoulli rates matrix, where the diagonal values are set to k_q/(2q).
  • − Erdös-Rényi model: with p nodes and k_q/2 edges.
  • Let ${\mathcal{M}}_d={\mathcal{M}}_{d,1}\cup {\mathcal{M}}_0$. For $(i,j)\notin {\mathcal{M}}_d$, generate S_d,l,i,j ~ Bernoulli(1, 0.3). For $(i,j)\in {\mathcal{M}}_d$, generate S_d,l,i,j ~ Bernoulli(1, r_d,i,j), where r_1,i,j is set to 0.5 with probability 0.1, and 0.8 otherwise, whereas r_2,i,j is set to 0.8 with probability 0.1, and 0.5 otherwise.

• + Bernoulli mixture: Generate ${\mathcal{M}}_d$ in the same way as before. Generate S_d,l,i,j ~ Bernoulli(1, r_d,i,j), where r_d,i,j = π_i,j * r_d,1,i,j + (1 − π_i,j) * r_d,2,i,j, with π_i,j ~ Uniform(0, 1). For $(i,j)\in {\mathcal{M}}_d$, r_d,1,i,j = r_d,2,i,j = 0.3, d = 1, 2. For $(i,j)\in {\mathcal{M}}_d$, r_1,1,i,j is set to 0.5 with probability 0.1, and 0.7 otherwise, whereas r_2,1,i,j is set to 0.7 with probability 0.1, and 0.5 otherwise, and r_d,2,i,j = r_d,1,i,j + 0.2.

• + Wishart with logarithm transformation: Select the sets ${\mathcal{M}}_{d,1}$ and ${\mathcal{M}}_0$ from q hypotheses, uniformly and randomly, with $|{\mathcal{M}}_{d,1}|=k_q/4$, and $|{\mathcal{M}}_0|=3k_q/4$, d = 1, 2. Let ${\mathcal{M}}_d={\mathcal{M}}_{d,1}\cup {\mathcal{M}}_0$. Generate Σ_d such that ${\Sigma }_{d,i,j}^{\prime }=\text{ Uniform }(3,5)$ if $(i,j)\in {\mathcal{M}}_d$ and ${\Sigma }_{d,i,j}^{\prime }=0$ otherwise. Let ${\Sigma }_{d,j,i}^{\prime }={\Sigma }_{d,i,j}^{\prime }$ and ${\Sigma }_d={\Sigma }_d^{\prime }+\left\{|{\lambda }_{\text{min}}\left({\Sigma }_d^{\prime }\right)|+0.5\right\}I$, where I is the identify matrix. Generate $S_{d,l}^{\prime }~\text{Wishart}\left(m^{-1}{\Sigma }_d,m\right)$, with m = 300, and $S_{d,l}=\text{log}\left[\text{round }\left\{\text{exp}\left(S_{d,l}^{\prime }\right)\right\}\right]$, where round(·) rounds a number to the nearest integer.

For each network structure, the parameter k_q controls the sparsity level, and we examine three levels, k_q = 0.2q, 0.15q and 0.1q, where q is the total number of the network links.

5.2. Results

First, we investigate the empirical size of the proposed global test Ψ_α for the global testing problem (1). For this problem, the population network means are equal to each other under the null hypothesis, and we set ${\mathcal{M}}_1={\mathcal{M}}_2$, and set the sample size n₁ = n₂ = 500. We also compare with the global testing method aSPU developed by Kim et al. (2014), implemented in the R package aSPU. Table 1 reports the empirical size of the two tests, in percentage, based on 1000 data replications under the significance level α = 5%. It is seen clearly from the table that our proposed global testing procedure controls the type I error reasonably well, while the aSPU method has a slight size inflation in some cases though not severe. In addition, we report the computation time of each method, in seconds, averaged over three sparsity levels and all replications. It is seen from the last two columns of the table that, the average computation time of aSPU is much longer than our method; e.g., for p = 100 and p = 200, it is about 9 times and 15 times of our method, respectively. This is because Kim et al. (2014) did not derive the theoretical null distribution of their test statistics, but instead employed the permutations to obtain the critical value, which results in a more time consuming procedure. We also note that, in this setting, the sample size is much smaller than the total number of hypotheses q, but is larger than the sample size we use in the multiple testing simulations. This is due to the relatively slow convergence rate of the Bernoulli normal approximation and the maximum type statistics.

Table 1:

The empirical size and computation time for our global test Ψ_α and the aSPU test of Kim et al. (2014). The empirical size is in percentage based on 1000 data replications. The computation time is in seconds averaged over three sparsity levels and all replications. The significance level is α = 5%, and the sample size is n₁ = n₂ = 500.

	Method	p = 100			p = 200			Computation time
		0.2q	0.15q	0.1q	0.2q	0.15q	0.1q	p = 100	p = 200
	Ψα	4.1	5.1	5.6	5.0	3.6	4.9	0.52	1.99
Power-law
	aSPU	5.8	6.7	6.0	5.8	5.6	5.7	4.86	32.6
	Ψα	4.3	5.4	5.2	5.9	4.4	5.1	0.56	2.00
Stochastic Block
	aSPU	6.2	6.1	5.5	6.2	7.0	5.4	4.66	32.8
	Ψα	5.3	5.4	4.1	4.8	5.4	5.4	0.52	2.01
Erdös-Rényi
	aSPU	5.9	7.5	5.2	6.1	5.3	6.4	4.74	32.3
	Ψα	6.2	5.1	4.9	4.8	3.4	5.5	0.51	2.06
Bernoulli mixture
	aSPU	6.8	6.3	5.4	7.0	5.4	6.0	4.77	32.9
	Ψα	4.9	4.5	5.7	4.2	4.8	4.5	0.54	2.23
Transformed Wishart
	aSPU	6.5	6.9	6.3	6.5	6.8	5.7	4.42	28.1

Next, we examine the empirical FDR and the empirical power of the simultaneous testing procedure for the multiple testing problem (2). We consider two sample sizes, n₁ = n₂ = 100 and n₁ = n₂ = 25, and the latter mimics the real data setting where the sample size is very limited. We apply both Algorithms 1 and 2, one with the proposed power enhancement, and one without. Table 2 reports the empirical FDR and power, both in percentage, based on 100 replications under the significance level α = 5% for the Bernoulli network structure. Table 3 reports the results for the Bernoulli mixture and Wishart with logarithm transformation. It is seen that, in all cases, the empirical FDRs are generally controlled under the nominal level by both algorithms. Algorithm 2 is slightly more conservative than Algorithm 1, which is mainly due to the normalization step of the weight calculation as shown in (5). A similar phenomenon has also been observed in Xia et al. (2019a). For the empirical power, it is seen that Algorithm 2 achieves a clear power improvement over Algorithm 1, without sacrificing the size of the test. This is mainly due to the utilization of the auxiliary information in Algorithm 2. Furthermore, the performance under the varying sample size confirms the power enhancement of Algorithm 2 as theoretically revealed in Section 4.4. We also observe that, the power gain becomes more substantial when the true difference s₁ − s₂ becomes more sparse, which agrees with our intuition explained in Section 4.1.

Table 2:

The empirical FDR and the empirical power for the simultaneous testing procedures, Algorithms 1 and 2. The results are in percentages based on 100 data replications. The significance level is α = 5%. The network structure is Bernoulli.

	p = 100						p = 200
	n1 = n2 = 100			n1 = n2 = 25			n1 = n2 = 100			n1 = n2 = 25
Network structure	0.2q	0.15q	0.1q	0.2q	0.15q	0.1q	0.2q	0.15q	0.1q	0.2q	0.15q	0.1q
Bernoulli, power-law	Empirical FDR
Algorithm 1	4.1	4.5	4.7	6.2	6.3	7.2	4.2	4.4	4.9	5.8	6.5	7.5
Algorithm 2	2.6	2.6	2.9	3.5	4.6	5.3	2.2	2.3	2.5	3.5	4.9	5.1
	Empirical power
Algorithm 1	88.7	87.0	84.7	42.2	40.8	39.7	88.7	87.0	84.9	41.6	40.5	39.9
Algorithm 2	92.1	91.7	90.9	54.8	54.1	53.4	92.3	91.8	91.0	54.2	53.9	53.2
Bernoulli, stochastic block	Empirical FDR
Algorithm 1	4.3	4.4	4.8	6.1	6.4	7.7	4.2	4.5	4.9	5.8	6.3	7.6
Algorithm 2	2.8	2.7	3.0	3.5	4.5	5.5	2.2	2.3	2.5	3.5	5.0	5.0
	Empirical power
Algorithm 1	89.0	87.1	84.8	41.5	40.4	40.0	89.0	87.0	84.9	41.4	40.4	40.1
Algorithm 2	92.2	91.7	90.8	54.5	54.5	54.0	92.5	91.9	90.9	54.2	53.9	53.4
Bernoulli, Erdös-Rényi	Empirical FDR
Algorithm 1	4.1	4.4	4.8	6.0	6.0	7.3	4.0	4.4	4.8	5.9	5.2	7.3
Algorithm 2	2.3	2.6	2.9	3.9	4.1	5.7	2.1	2.2	2.4	3.7	4.5	5.1
	Empirical power
Algorithm 1	88.0	86.9	84.7	44.1	41.8	40.6	88.1	86.8	84.5	44.4	42.0	40.8
Algorithm 2	91.8	91.3	90.6	54.7	54.4	53.3	91.9	91.4	90.5	54.6	54.1	53.6

Table 3:

The empirical FDR and the empirical power for the simultaneous testing procedures, Algorithms 1 and 2. The results are in percentages based on 100 data replications. The significance level is α = 5%. The network structure is Bernoulli mixture and transformed Wishart.

	p = 100						p = 200
	n1 = n2 = 100			n1 = n2 = 25			n1 = n2 = 100			n1 = n2 = 25
Network structure	0.2q	0.15q	0.1q	0.2q	0.15q	0.1q	0.2q	0.15q	0.1q	0.2q	0.l5q	0.1q
Bernoulli mixture	Empirical FDR
Algorithm 1	4.0	4.4	4.8	6.1	6.0	7.4	4.0	4.4	4.8	6.0	5.8	7.5
Algorithm 2	1.4	1.7	2.0	3.2	4.2	5.0	1.3	1.5	1.7	2.9	4.3	4.5
	Empirical power
Algorithm 1	88.3	87.2	85.6	41.8	40.8	41.1	88.2	87.1	85.7	41.6	40.7	41.2
Algorithm 2	93.8	93.6	93.6	54.2	54.3	54.1	93.5	93.6	93.5	53.9	54.2	54.1
Transformed Wishart	Empirical FDR
Algorithm 1	4.2	4.6	5.1	5.0	5.6	6.6	4.2	4.6	4.9	4.9	5.3	5.9
Algorithm 2	1.6	1.8	2.0	2.4	2.8	3.7	1.6	1.9	2.0	1.8	2.1	2.6
	Empirical power
Algorithm 1	63.5	65.9	69.6	44.1	46.8	50.6	52.6	55.7	60.4	37.5	40.2	43.1
Algorithm 2	70.9	73.8	78.4	50.3	54.5	59.9	59.8	63.9	69.9	41.4	45.2	50.0

6. Brain Connectivity Analysis

We illustrate our method with two brain connectivity analysis examples.

6.1. Structural connectivity analysis

The first example is a brain structural connectivity analysis of diffusion tensor images (DTI). DTI is a magnetic resonance imaging technique that measures the diffusion of water molecules to map white matter tractography in the brain. The data we analyze is the KKI-42 dataset, available at http://openconnecto.me/data/public/MR/archive/, and its detailed description can be found in Landman et al. (2011). This data consists of 21 subjects with no history of neurological conditions, aging from 22 to 61 years old. Each subject received two resting-state DTI under a scan-rescan imaging session. For simplicity, we treat the data as if those images were from independent samples, which is common for the analysis of this dataset (Wang et al., 2017). It results in a total sample size of 42 for this study. Brain regions are constructed following the Desikan Atlas (Desikan et al., 2006), leading to p = 68 regions equally divided in the left and right hemispheres. Each DTI image has been preprocessed and summarized in the form of a 68 × 68 network, where the edges record the total number of white matter fibers between the pair of nodes. It is also equally common to focus on the form of a binary network, where the edges become the binary indicators of presence or absence of white matter fibers (Wang et al., 2017). We partition the subjects into two age groups, the ones whose are younger than 30 years, and the ones who are 30 or older. Age 30 is a transition period, usually known as the “age 30 transition”, when the first phase of early adulthood comes to a close, and the basis for the next life structure is formed. Moreover, this partition yields about the same number of subjects for each group, with n₁ = 22 for the younger-than-30 age group, and n₂ = 20 for the older-than-30 age group. We study the age-related difference in structural connectivity patterns, which is of universal interest, as aging is the main risk factor for progressive loss of both structures and functions of brain neurons (Morrison and Hof, 1997).

We apply both multiple testing procedures Algorithms 1 and 2 to this dataset, first the binary network, then the count network with a logarithm transformation. We set the significance level at 0.05. For the binary network, out of the total of 2278 links, Algorithm 1 identifies 2 significantly different links, whereas the power-enhanced Algorithm 2 identifies 8 links, including the first link found by Algorithm 1 plus 7 additional links. For the count network, Algorithm 1 identifies 4 significantly different links, whereas Algorithm 2 identifies 15 links, including all the links found by Algorithm 1 plus 11 additional links. These results agree with both our theory and simulations, in that Algorithm 2 is usually able to recognize more significant links than Algorithm 1. Table 4 reports the identified links by the two algorithms for both types of network data. Some links found by our power-enhanced procedure agree with the neuroscience literature; for instance, the link between left fusiform and left temporal pole under the count network. The temporal pole, also known as Brodmanna area 38, is a paralimbic region involved in high-level semantic memories and socio-emotional processing. The fusiform gyrus is part of the temporal lobe and occipital lobe in Brodmann area 37, and is linked with various neural pathways related to recognition. Li et al. (2013) also found significant difference in structural connectivity patterns between the left fusiform and the left temporal pole, for the young subjects (18 to 23 years old) versus the middle-aged and old subjects (30 to 58, and 61 to 89 years old). Meanwhile, other links found by our procedure require further scientific validation; for instance the links between left temporal pole and orbitofrontal cortex. The latter is a prefrontal cortex region in the frontal lobe of the brain involved in the cognitive process of decision-making.

Table 4:

Structural connectivity analysis of the KKI-42 dataset. Reported are the significantly different links found by Algorithms 1 and 2 for the binary and count network data, respectively.

Binary network		Count network
Algorithm 1	Algorithm 2	Algorithm 1	Algorithm 2
r.posteriorcingulate ↔ l.superiorparietal	r.posteriorcingulate ↔ l.superiorparietal	r.corpuscallosum ↔ l.superiorparietal	r.corpuscallosum ↔ l.superiorparietal
r.posteriorcingulate ↔ l.supramarginal	r.precuneus ↔ l.postcentral	l.isthmuscingulate ↔ l.posteriorcingulate	l.isthmuscingulate ↔ l.posteriorcingulate
	r.caudalanteriorcingulate ↔ r.lingual	r.caudalmiddlefrontal ↔ r.rostralmiddlefrontal	r.caudalmiddlefrontal ↔ r.rostralmiddlefrontal
-	r.posteriorcingulate ↔ l.caudalmiddlefrontal	l.lateralorbitofrontal ↔ l.superiorfrontal	l.lateralorbitofrontal ↔ l.superiorfrontal
-	1. lateral occipital ↔ l.parsopercularis	-	r.posteriorcingulate ↔ l.precuneus
	r.superiorparietal ↔ l.precentral	-	r.caudalmiddlefrontal ↔ r.parstriangularis
-	r.paracentral ↔- l.superiorparietal	-	1.fusiform ↔ l.temporalpole
-	l.bankssts ↔ l.frontalpole	-	l.entorhinal ↔ l.lateralorbitofrontal
		-	r.corpuscallosum ↔ l.precuneus
		-	l.caudalmiddlefrontal ↔ l.pericalcarine
		-	r.bankssts ↔ r.postcentral
		-	l.lateralorbitofrontal ↔ l.temporalpole
		-	l.parsopercularis ↔ l.rostralmiddlefrontal
		-	l.medialorbitofrontal ↔ l.temporalpole
		-	l.corpuscallosum ↔ l.superiorparietal

6.2. Functional connectivity analysis

The second example is a brain functional connectivity analysis of functional magnetic resonance images (fMRI). Functional MRI measures the blood oxygen level signals, and provides a tool to study brain functional connectivity network. The data we analyze is the ADHD-200 dataset, and is available at http://neurobureau.projects.nitrc.org/ADHD200/Data.html. More detailed description can be found in Ahn et al. (2015). Attention deficit hyperactivity disorder (ADHD) is one of the most commonly diagnosed child-onset neurodevelopmental disorders, and has an estimated childhood prevalence of 5−10% worldwide (Pelham et al., 2007). This data consists of 96 subjects with ADHD, and 91 normal controls. Each subject received a resting-state fMRI scan, and each brain image is parcellated using the Anatomical Automatic Labeling (AAL) Atlas with p = 116 regions (Tzourio-Mazoyer et al., 2002). The resulting data is a spatial by temporal matrix, which is then turned into a Pearson correlation matrix or a partial correlation matrix to represent the brain functional connectivity network. Actually, both correlation measures are frequently used in functional connectivity analysis (Bullmore and Sporns, 2009). We use both measures to study the difference in functional connectivity patterns between the two groups of subjects with and without ADHD.

We again apply both multiple testing procedures Algorithms 1 and 2 to this dataset, first the Pearson correlation network, then the partial correlation network. For the Pearson correlation network, Algorithm 1 identifies no significantly different links, whereas the power-enhanced Algorithm 2 identifies 3 links. For the partial correlation network, Algorithm 1 again identifies no significantly different links, whereas the power-enhanced Algorithm 2 identifies 5 links. Table 5 reports the identified links by the two algorithms for both types of network data. One brain region that differentiating links concentrate is cerebellum. The cerebellum is responsible for motor control and cognitive functions such as attention and language, and dysfunction in the cerebellum in ADHD patients have been reported (Toplak et al., 2006). We also remark that, there are fewer links found here compared to those found in Xia and Li (2019). This is because the data in the format of spatial temporal matrix analyzed in Xia and Li (2019) carries more information than the data in the format of correlation matrix. Nevertheless, the focus of this article is to develop inferential tests for the scientific applications where only the data format of some symmetric network matrix is available.

Table 5:

Functional connectivity analysis of the ADHD-200 dataset. Reported are the significantly different links found by Algorithms 1 and 2 for the Pearson correlation and partial correlation network data, respectively.

Pearson correlation network		Partial correlation network
Algorithm 1	Algorithm 2	Algorithm 1	Algorithm 2
-	r.frontal.sup ↔ r.frontal.med.orb	-	l.paracentral.lobule ↔ r.paracentral.lobule
-	r.cerebelum6 ↔ r.cerebelum.8	-	r.frontal.sup.orb ↔ r.frontal.mid.orb
-	I.cerebelum8 ↔ vermis7	-	r.frontal.inf.orb ↔ 1.temporal.pole.sup
		-	r.fusiform ↔ r.cerebelum6
		-	l.frontal.sup ↔ l.frontal.mid

7. Conclusion

In this article, we develop both global and simultaneous inference methods for network comparisons when the data are observed in the form of p × p matrices, each of which encodes the network structure for an individual subject. This data format is different from those studied in the existing network literature, and leads to a different set of testing procedures and the associated theory. In addition, we propose a power enhancement approach to tackle the challenge of limited sample size in numerous applications.

We have primarily focused on the scenario of using a symmetric matrix to encode a network structure in this article. In principle, our methods can be extended to the asymmetric matrix scenario as well, with corresponding modifications of the total number of tests and the related theoretical properties. In the interest of space, we leave it as our future research.