On the fast small-sample kernel independence test for microbiome community-level association analysis

Summary

Zhan et al. (2017) presented a kernel RV coefficient (KRV) test to evaluate the overall association between host gene expression and microbiome composition, and showed its competitive performance compared to existing methods. In this paper, we clarify the close relation of KRV to the existing generalized RV (GRV) coefficient, and show that KRV and GRV have very similar performance. Although the KRV test could control the type I error rate well at 1% and 5% levels, we show that it could largely under-estimate p-values at small significance levels leading to significantly inflated type I errors. As a partial remedy, we propose an alternative p-value calculation, which is efficient and more accurate than KRV p-value at small significance levels. We recommend that small KRV test p-values should always be accompanied and verified by the permutation p-value in practice. In addition we analytically show that KRV can be written as a form of correlation coefficient, which can dramatically expedite its computation and make permutation p-value calculation more efficient.

1 Introduction

Zhan et al. (2017) proposed a kernel RV (KRV) coefficient to test the overall association between host gene expression and microbiome composition. KRV is closely related to the generalized RV (GRV) coefficient of Minas et al. (2013), and both are broadly applicable to test the association between two sets of multivariate outcomes, X and Y, measured on a same cohort of n subjects. Both are essentially the correlations between two similarity matrices: KRV uses kernel matrix and GRV uses Gower matrix (Gower, 1966) respectively. The similarity matrix is typically computed from some distance matrix. Specifically given two n × n distance matrices D_x and D_y computed from X and Y respectively, we compute the Gower matrices

$$G_x=-\frac{1}{2}H{D}_x^{2}H,G_y=-\frac{1}{2}H{D}_y^{2}H,H={\mathit{I}}_n-{\mathit{J}}_n/n,$$

where I_n is the n-th order identity matrix, J_n is a n × n matrix of ones, $D_x^2$ and $D_y^2$ are the element-wise matrix squares. GRV can be computed as

$$Q_g=\frac{tr(G_x{G}_y)}{\sqrt{tr(G_x{G}_x)}\sqrt{tr(G_y{G}_y)}},$$

where tr() denotes the matrix trace. To compute KRV, we first transform the Gower matrix G into a kernel matrix K following the approach of Zhao et al. (2015). Specifically given the eigen decomposition G = UΔU^T with Δ = diag(λ₁, · · ·, λ_n) being the eigen values, we calculate K = U Δ̃U^T, where Δ̃ = diag(|λ₁|, · · ·, |λ_n|). Let K_x and K_y be the transformed kernel matrices from G_x and G_y respectively. The KRV is then computed as

$$Q_k=\frac{tr({\stackrel{\sim }{K}}_x{\stackrel{\sim }{K}}_y)}{\sqrt{tr({\stackrel{\sim }{K}}_x{\stackrel{\sim }{K}}_x)}\sqrt{tr({\stackrel{\sim }{K}}_y{\stackrel{\sim }{K}}_y)}},$$

where K̃_x = HK_xH and K̃_y = HK_yH are the centered kernel matrices. Here the kernel matrix is essentially the Gower matrix with a positive semi-definiteness correction. In this spirit, the KRV is equivalent to the GRV. So when the kernel matrix is directly available (not computed from some distance matrix), we can substitute the centered kernel matrix as Gower matrix to calculate GRV (in section 3, we will see one such example). For most commonly used and well-defined distances, the Gower matrix is typically positive-definite, and GRV and KRV are identical. In our numerical studies, we have observed that GRV and KRV give very similar results.

2 Computation

We first show that the RV-type statistic can be written as a sample correlation. Given two symmetric and centered n × n matrices A = (a_ij), B = (b_ij), we can easily check that tr(AB) = Σ_i,j a_ijb_ij, $tr(AA)={\sum }_{i,j}{a}_{ij}^2,tr(BB)={\sum }_{i,j}{b}_{ij}^2$. Hence

$$Q=\frac{tr(AB)}{\sqrt{tr(AA)tr(BB)}}=\frac{{\sum }_{i,j}{a}_{ij}{b}_{ij}}{\sqrt{{\sum }_{i,j}{a}_{ij}^2}\sqrt{{\sum }_{i,j}{b}_{ij}^2}}.$$

Therefore Q = cor[vec(A), vec(B)], where vec() is the vector operator that stacks the columns of a matrix into a vector.

When both A and B are positive semi-definite kernel matrices, denote the Cholesky decompositions $A/\sqrt{tr(AA)}=C{C}^T,B/\sqrt{tr(BB)}=D{D}^T$, where C and D are n×n lower-triangular matrices. Denote Z = D^TC = (z_ij), which can be treated as a matrix of Score type statistics testing the association between two sets of “working outcomes”: the n columns of C and D. We then have $Q=tr[{(D^{T}C)}^T(D^{T}C)]={\sum }_{i,j}{z}_{ij}^2\ge 0$, which is closely related to the multi-trait variant set association test statistic (Wu and Pankow, 2016).

2.1 Permutation p-value calculation

The distributions of GRV and KRV are generally unknown, and permutations have been typically used to obtain their p-values (Minas et al., 2013; Zhan et al., 2017). Specifically their null distributions are obtained by randomly shuffling rows and columns of one kernel matrix simultaneously. Note that randomly shuffling the columns and rows of B simultaneously does not change tr(BB). Therefore computationally we just need to calculate vec(A)^T vec(B) under permutation to compute the p-value of Q, which can be done very efficiently.

2.2 Permutation distribution approximation

To further improve the computation efficiency, an approximate Gamma distribution is commonly used to compute the p-value analytically for the RV-type statistics. The approximate Gamma distribution is estimated by matching the first three moments with the permuted RV-type statistics.

Denote the observed RV statistic as Q, and its mean, standard error, and skewness under permutation as (μ, σ, γ), where μ = E(Q), $\sigma =\sqrt{E[{(Q-\mu )}^2]}$, and γ = E[(Q − μ)³/σ³]. Here the expectation is with respect to the permutation distribution of Q. We approximate the permutation null distribution of sign(γ)(Q−λ) with a Gamma(k, θ) distribution, where k is the shape, θ is the scale and

$$k=4/{\gamma }^2,\theta =\sigma \mid \gamma \mid /2,\lambda =\mu -2\sigma /\gamma .$$

By matching the first three moments, the approximated Gamma distribution has been shown to control type I errors well at the 0.05 significance level.

For small p-values, the accuracy of approximation critically depends on the tail of Gamma distribution. In our numerical studies, we have found that the Gamma(k, θ) approximation could largely under-estimate small p-values (leading to significantly inflated type I errors at small significance levels). For example, when analyzing the association of host gene expression and microbiome profiles in 196 pre-pouch ileum (PPI) samples using KRV test (see Section 3), the Gamma approximation gives a p-value of 6.5 × 10⁻⁵, while the actual permutation p-value is 1.5 × 10⁻⁴ (based on 10⁷ permutations).

To improve the performance of Gamma approximation, we propose to match kurtosis instead of skewness as follows. Denote ν = E[(Q − μ)⁴/σ⁴] as the kurtosis under permutation, then we approximate the permutation null distribution of sign(ν − 3)(Q − λ̃) with a Gamma(k̃, θ̃) distribution, where

$$\stackrel{\sim }{k}=6/\mid \nu -3\mid ,\stackrel{\sim }{\theta }=\sigma \sqrt{\mid \nu -3\mid /6},\stackrel{\sim }{\lambda }=\mu -\sigma \sqrt{6\mid \nu -3\mid }/(\nu -3).$$

We can easily verify that it leads to the same mean, variance and kurtosis as permuted RV statistics. This new Gamma approximation gives a p-value of 1.3 × 10⁻⁴ for the 196 PPI samples.

3 Numerical studies

3.1 Application to PPI study

For illustration, we reanalyze the association of host gene expression and microbiome profiles in 196 pre-pouch ileum (PPI) samples (Morgan et al., 2015). Specifically 7000 OTU counts measured by 16S rRNA and around 20,000 microarray gene expressions were analyzed following Zhan et al. (2017).

For the host transcript, we calculate a Gaussian kernel, K_ij = exp(−||Y_i−Y_j||²/σ²), where Y_i denotes the centered expression values for the i-th sample, and σ² is taken as the median of all pairwise distances. For the OTU data, we computed the Bray-Curtis distance kernel (denoted as K_BC) directly from the counts, and the following five UniFrac-type distance kernels based on a trained phylogenetic tree (Li, 2015; Zhan et al., 2017): (1) weighted UniFrac distance kernel (denoted as K_w); (2) unweighted UniFrac distance kernel (denoted as K_u); (3) variance adjusted weighted UniFrac distance kernel (denoted as K_v); (4) generalized UniFrac distance kernel K_θ with θ = 0 (denoted as K₀); (5) generalized UniFrac distance kernel K_θ with θ = 0.5 (denoted as K_0.5). We apply the KRV, GRV, and the proposed method (denoted as “RRV” for revised RV test). As a benchmark, we also included the permutation approach based on 10⁷ random permutations (denoted as “Perm”), which can be treated as the gold standard. Table 1 shows the results. Overall, we can see that KRV and GRV have very similar performance. KRV generally under-estimates the significance p-values compared to the permutation approach. The proposed RRV mitigates the inflation of KRV, and is closer to the permutation p-values.

Table 1.

P-values of testing around 20,000 gene expressions and microbiome association in 196 PPI samples. Permutation p-values (denoted “Perm”) are obtained based on 10⁷ Monte Carlo permutations, and can be treated as the gold standard. The proposed method is denoted “RRV” (for revised RV test).

Test	Kw	Ku	K0.5	KBC	Kv	K0
Perm	1.23 × 10−2	1.46 × 10−4	5.12 × 10−3	7.82 × 10−3	1.96 × 10−2	4.58 × 10−4
KRV	1.20 × 10−2	6.45 × 10−5	4.64 × 10−3	7.44 × 10−3	1.97 × 10−2	2.57 × 10−4
GRV	1.22 × 10−2	6.45 × 10−5	4.43 × 10−3	8.09 × 10−3	2.03 × 10−2	2.57 × 10−4
RRV	1.34 × 10−2	1.26 × 10−4	5.72 × 10−3	8.46 × 10−3	2.20 × 10−3	4.75 × 10−4

3.2 Simulation study

For illustration, we fix the six microbiome kernels we computed previously for the 196 PPI samples, and randomly simulate the expressions of 100 genes for these 196 samples from a multivariate normal distribution with zero means and compound symmetry covariance matrix with unit variance and pairwise covariance being 0.25. We compute the Gaussian kernel for these gene expression profiles as previously. We consider three tests: KRV, the proposed RRV based on matching kurtosis, and the permutation test based on randomly shuffling the columns and rows of the expression Gaussian kernel. Permutation p-values (denoted “Perm”) are obtained adaptively as follows. We first used 10⁵ random permutations to estimate p-values. For those p-values less than 10⁻³, we then increased the permutations to 10⁶ to obtain more accurate estimates. We conducted 10⁷ simulations to estimate the type I errors at significance level α = 0.05, 10⁻², 10⁻³, 10⁻⁴, 10⁻⁵. The results are summarized at Table 2. Overall we can see that the permutation approach very accurately controls the type I errors. The KRV test has increased inflations at more stringent significance levels. The proposed RRV has reduced inflations compared to KRV, though it still has increased inflations at more stringent significance levels.

Table 2.

Ratio of empirical type I errors divided by the significance level α estimated over 10⁷ simulations. Listed within parenthesis are the standard errors divided by α. Permutation p-values (denoted “Perm”) are obtained adaptively as follows. We first used 10⁵ Monte Carlo simulations to estimate p-values. For those p-values less than 10⁻³, we then used 10⁶ Monte Carlo permutations to obtain more accurate estimates.

	Kw					Ku
α	0.05	10−2	10−3	10−4	10−5	0.05	10−2	10−3	10−4	10−5
KRV	0.94 (0.001)	1.04 (0.003)	1.36 (0.01)	1.86 (0.04)	2.92 (0.17)	0.92 (0.001)	1.05 (0.003)	1.49 (0.01)	2.41 (0.05)	4.12 (0.20)
RRV	0.93 (0.001)	0.93 (0.003)	1.07 (0.01)	1.25 (0.04)	1.68 (0.13)	0.90 (0.001)	0.91 (0.003)	1.06 (0.01)	1.38 (0.04)	1.66 (0.13)
Perm	1.00 (0.001)	1.00 (0.003)	1.02 (0.01)	0.99 (0.03)	0.99 (0.10)	1.00 (0.001)	1.00 (0.003)	1.00 (0.01)	1.02 (0.03)	0.99 (0.10)
	K0.5					KBC
α	0.05	10−2	10−3	10−4	10−5	0.05	10−2	10−3	10−4	10−5
KRV	0.93 (0.001)	1.05 (0.003)	1.44 (0.01)	2.15 (0.05)	3.50 (0.19)	0.95 (0.001)	1.05 (0.003)	1.39 (0.01)	2.01 (0.04)	3.10 (0.18)
RRV	0.92 (0.001)	0.92 (0.003)	1.06 (0.01)	1.34 (0.04)	1.86 (0.14)	0.93 (0.001)	0.92 (0.003)	1.02 (0.01)	1.23 (0.04)	1.52 (0.12)
Perm	1.00 (0.001)	1.00 (0.003)	1.01 (0.01)	1.01 (0.03)	0.98 (0.10)	1.00 (0.001)	1.00 (0.003)	1.00 (0.01)	1.02 (0.03)	0.98 (0.10)
	Kv					K0
α	0.05	10−2	10−3	10−4	10−5	0.05	10−2	10−3	10−4	10−5
KRV	0.92 (0.001)	1.03 (0.003)	1.44 (0.01)	2.23 (0.05)	3.62 (0.19)	0.92 (0.001)	1.06 (0.003)	1.60 (0.01)	2.80 (0.05)	5.10 (0.23)
RRV	0.90 (0.001)	0.91 (0.003)	1.06 (0.01)	1.33 (0.04)	1.83 (0.14)	0.89 (0.001)	0.90 (0.003)	1.06 (0.01)	1.46 (0.04)	2.09 (0.14)
Perm	1.00 (0.001)	1.00 (0.003)	0.99 (0.01)	0.97 (0.03)	1.01 (0.10)	1.00 (0.001)	1.00 (0.003)	1.00 (0.01)	1.01 (0.03)	1.00 (0.10)

Figure 1 shows the QQ-plots for the KRV and RRV p-values based on the K_w and K₀ kernels. We can clearly see the trend of increasing inflations at more stringent significance levels.

Figure 1.

QQ plots: the dashed line is the diagonal line. We have plotted the p-values on the log10 scale for better visualization.

We also investigated the performance of different methods under linear kernel and with different dependence structures. Similar results are observed consistently across all simulation scenarios. The complete results (QQ plots and type I errors at various α levels) are available at the supplementary materials.

4 Discussion

Another closely related and widely used approach in ecology is the Mantel test (Mantel, 1967). It essentially calculates the correlation between two distance matrices, in contrast to the Gower/Kernel (roughly squared distance) matrices used in the RV statistics. In the special case of using linear kernel based on a single variable, the RV type statistic reduces to the Pearson correlation coefficient, while the Mantel test does not. In our numerical studies, the Mantel test performs worse than the RV statistics (see supplementary materials for results), which are consistent with the simulation studies of Minas et al. (2013).

When assessing the significance of RV-type statistics, we need to be cautious when using the existing Gamma approximation based p-values. In light of their simple structure as a correlation and the common practice of comparing different methods based on the computed p-values, we recommend that small Gamma approximation p-values should always be verified by the actual permutation p-values.