A regularized Hotelling’s T² test for pathway analysis in proteomic studies

Abstract

Recent proteomic studies have identified proteins related to specific phenotypes. In addition to marginal association analysis for individual proteins, analyzing pathways (functionally related sets of proteins) may yield additional valuable insights. Identifying pathways that differ between phenotypes can be conceptualized as a multivariate hypothesis testing problem: whether the mean vector μ of a p-dimensional random vector X is μ₀. Proteins within the same biological pathway may correlate with one another in a complicated way, and type I error rates can be inflated if such correlations are incorrectly assumed to be absent. The inflation tends to be more pronounced when the sample size is very small or there is a large amount of missingness in the data, as is frequently the case in proteomic discovery studies. To tackle these challenges, we propose a regularized Hotelling’s T² (RHT) statistic together with a non-parametric testing procedure, which effectively controls the type I error rate and maintains good power in the presence of complex correlation structures and missing data patterns. We investigate asymptotic properties of the RHT statistic under pertinent assumptions and compare the test performance with four existing methods through simulation examples. We apply the RHT test to a hormone therapy proteomics data set, and identify several interesting biological pathways for which blood serum concentrations changed following hormone therapy initiation.

1 INTRODUCTION

Proteins participate in virtually every process within cells, and they are essential parts of organisms. Proteomics technology allow us to measure protein abundance in a high dimensional fashion. Though antibody array-based approaches could conceptually generate data analogous to microarray data, such platforms are still at an early stage of development. Mass spectrometry(MS)-based platforms remain the workhorse for proteomic research. The resulting data could provide insight into biological systems relevant to a disease or treatment and lead to the identification of related mediating variables or biomarkers.

Proteins work independently and interactively to perform various biological functions. Thus, in addition to association analyses for individual proteins, there is an interest in pathway analyses. Here, pathways refer to sets of proteins that are relevant to specific biological functions, without regard to the state of knowledge concerning the interplay among such proteins. Pathway analyses aim to detect a priori defined sets of proteins that are associated with phenotypes of interest. Several informatics databases (Ashburner et al. 2000; Kanehisa 2002; Subramanian et al. 2005) provide protein functional pathway classifications. Pathway analyses complement the marginal analyses for individual proteins and support further exploration of information in the data. Proteins in each pathway are related by a priori information on biological function, and thus they may lead to valuable insight into the disease etiology or treatment effect and could inform clinical decisions concerning disease prevention or therapeutic maneuvers. Furthermore, when multiple proteins from a same pathway show concerted signals, there may be enhanced power for detecting a group association, compared to that for each separately.

Various methods have been proposed for pathway analysis (or gene set analysis) in microarray studies (Mootha et al. 2003; Subramanian et al. 2005; Tian et al. 2005; Dinu et al. 2007; Efron and Tibshirani 2007). In other related works, Lu et al. (2005) proposed a multiple forward search algorithm for selecting a subset of genes whose expressions differ most between groups; Shojaie and Michailidis (2009) made explicit usage of the regulatory relationships among genes in pathways to detect differentially expressed subnetworks; and Wu et al. (2009) implemented a sparse linear-discriminant-analysis method to test the significance of pathways and at the same time to select subsets of genes that drive the significant pathway effect. More recently, pathway analysis has also been pursued in genome-wide association studies (Wang et al. 2007; Chen et al. 2010).

However, studies of high-dimensional protein expression typically involve quite different technologies than do microarray or genetic association studies, with corresponding major differences in pertinent data analysis methods. Specifically, since proteins of interest may be quite large, MS platforms typically enzymatically digest proteins into peptides, and identify peptides by peaks at their molecular mass in a mass spectrum, with peptide concentration proportional to peak size. Because of run-to-run variations in peak sizes from the same specimen, some of the stronger proteomic platforms focus on concentration ratios for samples to be compared, for example, pre- versus post-treatment specimens, from the same study subject, or specimens from cases which developed a study disease versus corresponding matched controls. Particularly, the members of such a pair are labeled with isotopes having known molecular weight and the ratio of peptide peak sizes, that are separated by the difference in molecular weight of the two labels, provide estimates of peptide concentration ratios. Such ratios, for a set of uniquely determining peptides then yield concentration ratio estimates for the proteins from which they arise. In addition, proteins may vary in abundance over several orders of magnitude, and specimens may be highly fractionated, prior to peptide digestion and liquid chromatography-tandem mass spectrometry within each fraction, to facilitate reliable protein identification. See Eckel-passow et al. (2009) for further detail on mass spectrometry-based proteomic assessment.

As a result of these rather complex assessment procedures, the proteomic profiles that motivate this work consist of estimated concentration ratios for a few hundred proteins. Since these determinations are time consuming and expensive, specimens from several study subjects are typically pooled prior to proteomic analysis, and the number of paired samples to be contrasted may be quite small (10 pooled samples in our application). Also, the mass spectrometer sampling of peaks is dynamic, so that there may be a concentration ratio estimate for a particular protein from one sample pair, but the corresponding data may be missing from the next pair. These features imply the need for novel data analytic methods for paired sample comparisons. Furthermore, proteins having related functions may well yield correlated concentration ratios and act in a concerted fashion (Alon et al. 1999), and ignoring such correlation could lead to inflated type I error rates (Tian et al. 2005; Dinu et al. 2007).

Denote the log 2 ratios between two phenotypes (for example, cases versus controls or baseline versus 1-year after treatment) for proteins in a pathway by X_p×n = {x₁,…, x_n}. Suppose {x_i} are n independently identically distributed samples from a multivariate distribution, whose mean is μ and covariance matrix is Σ_p×p, consider a test of the hypothesis, H₀ : μ = μ₀. A classical tool to this multivariate hypothesis testing problem is the Hotelling’s T² (HT) test. The HT statistic is given by

$$\mathrm{H}\mathrm{T}=n{({\mathbf{X̅}}_n-{\mathit{\mu }}_{\mathbf{0}})}^T{\mathbf{S}}_n^{-1}({\mathbf{X̅}}_n-{\mathit{\mu }}_{\mathbf{0}}),$$ (1)

where ${\mathbf{X̄}}_n=\frac{1}{n}\sum _{i=1}^n{\mathit{x}}_{\mathit{i}}$ is the sample mean vector, and ${\mathbf{S}}_n=\frac{1}{m}\sum _{i=1}^n({\mathit{x}}_{\mathit{i}}-{\mathbf{X̅}}_n){({\mathit{x}}_{\mathit{i}}-{\mathbf{X̅}}_n)}^T$ is the sample covariance matrix (m = n − 1). Under a multivariate normality assumption, when p < n, the HT statistic is invariant under linear transformations and the distribution of this test statistic does not depend on the covariance matrix Σ. Under the null, $\frac{m-p+1}{pm}$HT follows a F_p,m−p+1 distribution. However, the HT test has very poor performance when p is close to n; and is ill-defined when p equals or exceeds n. This is because the test statistic relies on the inverse of sample covariance matrix $({\mathbf{S}}_n^{-1})$, which has large variation when p is close to n and does not exist when p ≥ n. To address this limitation, Dempster (1958) and Dempster (1960) proposed a non-exact test based on a χ² approximation. Bai and Saranadasa (1996) proposed a simplified test that has equivalent asymptotic power to Dempster’s test. Srivastava (2007) proposed a generalized Hotelling’s T² statistic, denoted as F⁺, which converges to a normal distribution with a suitable standardization. A few other test statistics have also been proposed for the p ≥ n scenario, including Fujikoshi et al. (2004), Bai and Silverstein (2004), Schott (2007), Srivastava and Du (2008), Pan and Zhou (2008), Liang and Tang (2009) and Chen and Qing (2010). In most of these methods, asymptotic approximations are used for deriving the significance levels of the test statistics under the null. However, when sample size is small and/or the data has a high rate of missingness, the null distributions of the test statistics may differ substantially from their asymptotic approximations. Hence, we instead use a non-parametric approach to derive the significance levels for the proposed test statistic.

In this paper, we propose a regularized Hotelling’s T² (RHT) test for both p < n and p ≥ n scenarios. The idea is to employ regularization techniques in the nonparametric testing context. Specifically, regularization is introduced to stabilize the inverse of the sample covariance matrix in (1) giving, for λ > 0,

$$\mathrm{R}\mathrm{H}\mathrm{T}=n{({\mathbf{X̅}}_n-{\mathit{\mu }}_{\mathbf{0}})}^T{({\mathbf{S}}_n+\lambda \mathbf{I})}^{-1}({\mathbf{X̅}}_n-{\mathit{\mu }}_{\mathbf{0}}).$$ (2)

Note, similar regularization procedures on the sample covariance matrix have been used in the literature in other contexts such as estimation and prediction (Lin and Perlman 1985; Ledoit and Wolf 2004; Schafer and Strimmer 2007). Compared to the test statistics proposed by Dempster (1958) and Dempster (1960), Bai and Saranadasa (1996) and the others mentioned earlier, RHT has a simpler form, and appears to be comparatively more robust to complex missing data patterns. Recently, Yates and Reimers (2009) independently proposed a similar test statistics to (2) under a rather different scenario, where the authors aim to perform two-sample pathway analysis in large microarray studies and derive the p-values based on permuting the group labels. However, permutation is not feasible in the proteomics studies we considered in this paper – the observed data are protein concentration ratios between two different groups and the sample sizes are usually extremely small (n = 10 in our example). Although for such one-sample (ratio of two groups) data, flipping the signs of ratios is analogous to permuting the group labels, this strategy for generating null distributions does not work under the small sample size setting, as we conferred through simulation studies. Hence, we propose a resampling scheme to estimate p-values, in which we implement an objective approach to selecting the regularization parameter. This testing procedure takes advantage of the asymptotic distribution theory for RHT which we establish in this manuscript. Specifically, we derive an asymptotic null distribution of the RHT test statistic as both n and p increase to infinity at a comparable rate, which then leads to an asymptotic cut-off value of the test that is completely data driven. We also quantify the power of the test and provide sufficient conditions for consistency of the RHT test. These theoretical results are helpful in understanding scenarios under which the RHT statistics may or may not perform well. The proposed testing procedure is specifically designed to handle limited sample sizes and missingness in the data, assuming missing completely at random (see Section 4 for the justification of this assumption in our application).

We now use a simple simulated example to further motivate the RHT statistic.

Example A. First, we set n = 10 and p = 8 to mimic the dimension of our proteomic study. We sample X from a multivariate normal distribution N(μ, Σ_p×p), and consider three different settings:

1. Independent Null: μ₁ = ⋯ = μ₈ = 0; Σ = I;

2. Dependent Null:μ₁ = ⋯ = μ₈ = 0; Σ = Σ₁ with σ_ii = 1 and σ_ij = 0.5 for i ≠ j;

3. Dependent Alternative: μ₁ = ⋯ = μ₄ = 1, μ₅ = ⋯ = μ₈ = 0; Σ = Σ₁.

Figure 1 shows the distribution of the following statistics: RHT, HT and the sum of absolute t-statistic (SAT), i.e. the sum of absolute values of t-statistics from each of the p variables (Ackermann and Strimmer 2009). In pathway analysis, because the true Σ_p×p is usually unknown, the Independent Null (IN, solid curves) obtained by resampling is often used to derive significance levels of test statistics. For SAT, since the correlation among the p variables is ignored, its distribution under IN is much more spread than its distribution under Dependent Null (DN), which may induce inflated type I error rate. On the other hand, for HT and RHT, the distributions under IN are either identical to or stochastically greater than that under DN, suggesting the type I error rate can be properly controlled. In addition, the greater the deviation between the null (dashed curves) and the alternative (dotted curves), the more powerful is the test. In this sense, HT has a larger overlap between the null and the alternative than RHT, suggesting lesser power compared to RHT. In Supplementary Information, we also examine the performance of a few other test statistics, including Dempster’s test, Bai and Saranadasa’s test, and the F⁺ test. These test statistics also suffer from inflated type I error rates, when resampling based procedures are used.

Figure 1.

Distribution of various test statistics on a simulated example with p = 8, N = 10. The dashed curves are the densities of the test statistics from Independent Null. The solid curves are the densities under Dependent Null. And the dotted curves are the densities under Dependent Alternative. Please see the text for more details.

The remainder of the paper is organized as follows: In Section 2, we present the test statistic, its asymptotic properties, and a non-parametric testing procedure. In Section 3, we compare the performance of RHT with four other existing test statistics on simulated examples. In Section 4, we apply the test to our motivating proteomic study and identify pathways related to postmenopausal hormone therapy. The theoretical proofs are provided in Section 5, and we conclude the paper in Section 6.

2 METHODS

2.1 RHT test statistics and its asymptotic properties

For ease of presentation, we assume μ₀ = 0. When p < n, Hotelling’s T² (HT) statistic $\mathrm{H}\mathrm{T}=n{\mathbf{X̅}}_n^T{\mathbf{S}}_n^{-1}{\mathbf{X̅}}_n$ is a classical tool for testing μ = 0. However, as noted above, HT has poor power when p is close to n, and is ill-defined when p ≥ n. To avoid these limitations, we impose regularization on the sample covariance matrix and propose a regularized Hotelling’s T² statistics (RHT)

$$\mathrm{R}\mathrm{H}\mathrm{T}(\lambda )=n{\mathbf{X̅}}_n^T{({\mathbf{S}}_n+\lambda {\mathbf{I}}_p)}^{-1}{\mathbf{X̅}}_n.$$ (3)

Here λ is a positive parameter. RHT can also be viewed as a union intersection test (UIT) statistic: For a ∈ R^p, define y_a = a^Tx ~ N(a^Tμ, a^TΣa). Denote the F-statistic for testing H_0a : E(y_a) = 0 as $t_{\mathbf{a}}=\frac{{\mathit{y̅}}_{\mathbf{a}}^2}{\widehat{\mathit{\text{var}}({\mathit{y}}_{\mathbf{a}})}}=\frac{{\mathbf{a}}^T{\mathbf{X̅}}_n{\mathbf{X̅}}_n^T\mathbf{a}}{{\mathbf{a}}^T{\mathbf{S}}_n\mathbf{a}}$. Then, HT = max_a≠0t_a, a UIT statistic for testing H₀ = ∩_a H_0a : μ = 0. If we replace the sample variance in the denominator of t_a with a^T (S_n + λI)a, a regularized estimate of the variance of y_a, we get another UIT statistic ${\text{max}}_{\mathbf{a}\ne 0}\frac{{\mathbf{a}}^T{\mathbf{X̅}}_n{\mathbf{X̅}}_n^T\mathbf{a}}{{\mathbf{a}}^T({\mathbf{S}}_n+\lambda \mathbf{I})\mathbf{a}}$, which is exactly our RHT.

In the presence of missingness, ${\mathbf{X̅}}_n={({\mathrm{x̄}}^k)}_{k=1}^p$ and ${\mathbf{S}}_n={((s^{kl}))}_{k,l=1}^p$ are defined as

$${\mathrm{x̄}}^k=\{\begin{array}{cc}\frac{{\sum }_{i=1}^n{x}_i^{k}I(x_i^k\ne NA)}{{\sum }_{i=1}^{n}I(x_i^k\ne \mathrm{N}\mathrm{A})},\hfill & \mathrm{i}\mathrm{f}{\sum }_{i=1}^{n}I(x_i^k\ne \mathrm{N}\mathrm{A})>0;\hfill \\ 0,\hfill & \mathrm{i}\mathrm{f}{\sum }_{i=1}^{n}I(x_i^k\ne \mathrm{N}\mathrm{A})=0.\hfill \end{array}{s}^{kl}=\{\begin{array}{cc}\frac{{\sum }_{i=1}^n(x_i^k-{\mathrm{x̄}}^k)(x_i^l-{\mathrm{x̄}}^l)I(x_i^k\ne \mathrm{N}\mathrm{A},x_i^l\ne \mathrm{N}\mathrm{A})}{{\sum }_{i=1}^{n}I(x_i^k\ne \mathrm{N}\mathrm{A},x_i^l\ne \mathrm{N}\mathrm{A})},\hfill & \mathrm{i}\mathrm{f}{\sum }_{i=1}^{n}I(x_i^k\ne \mathrm{N}\mathrm{A},x_i^l\ne \mathrm{N}\mathrm{A})>0;\hfill \\ 0,\hfill & \mathrm{i}\mathrm{f}{\sum }_{i=1}^{n}I(x_i^k\ne \mathrm{N}\mathrm{A},x_i^l\ne \mathrm{N}\mathrm{A})=0,\hfill \end{array}$$

where NA denotes ‘not measured’. We assume here that observations are missing completely at random (Little and Rubin 1987).

The ridge-like regularization in (3) resolves the singularity problem for computing ${\mathbf{S}}_n^{-1}$, so that RHT can be calculated for both p < n and p ≥ n cases. The regularization also helps to stabilize the inverse of the sample covariance matrix, thus RHT is expected to enjoy superior power compared to HT when p is close to n. We illustrate this point by characterizing the asymptotic power of the two test statistics under pertinent assumptions.

Suppose that {x_i} ~_i.i.d. N(μ, Σ_p) and there is no missingness. We make the following assumptions:

• A1: Σ_p is a p × p positive definite matrix.

• A2: p, n → ∞ such that p/n → γ ∈ (0, ∞) and $\sqrt{n}|p/n-\gamma |\to 0$.

• A3: Let τ_1,p ≥ ⋯ ≥ τ_p,p > 0 be the eigenvalues of Σ_p. Then the empirical spectral distribution (ESD) of Σ_p, defined by $H_p(\tau )=\frac{1}{p}{\sum }_{j=1}^p{\mathbf{1}}_{[{\tau }_{j,p},\infty )}(\tau )$, converges as p → ∞ to a probability distribution function H(τ) at every point of continuity of H, whose support, supp(H), is included in a compact set [h₁, h₂] with 0 < h₁ ≤ h₂ < ∞.

• A4: lim sup_p→∞ τ_1,p < ∞ and lim inf_p→∞ τ_p,p > 0.

Note that A3 does not imply A4, since there can be a small number (o(p), to be precise) of eigenvalues of Σ_p that can either diverge to infinity or converge to zero, even though A3 is satisfied.

Variants of conditions A1–A4 constitute fairly standard regularity conditions in the context of analysis of the empirical spectral distribution (ESD) of a high-dimensional sample covariance matrix. Conditions A1 and A4 appear, e.g. in Bai and Silverstein (2004) and Karoui (2007). Our setting is most closely related to the problem studied in Ledoit and Péché (2010) and except for the additional requirement that $\sqrt{n}|p/n-\gamma |\to 0$ (needed for proving the asymptotic normality of the RHT statistic), our conditions match with conditions H2 to H4 in that paper.

With assumptions A1–A4, we derive Theorem 1 to characterize the asymptotic distribution of RHT(λ) under the null. First, we introduce some notations used in the theorem. For z ∈ ℂ, denote m_F (z) as the Stieltjes transform of the limiting ESD of S_n, viz. $F_{n,p}(x)=\frac{1}{p}{\sum }_{j=1}^p{\mathbf{1}}_{[d_j,\infty )}(x)$, where d₁ ≥ ⋯ ≥ d_p are the eigenvalues of S_n. In other words, m_F (z) is the pointwise a.s. limit of

$$m_{F_{n,p}}(z)=\frac{1}{p}\sum _{j=1}^p\frac{1}{d_j-z}=\frac{1}{p}\text{tr}({({\mathbf{S}}_n-z{\mathbf{I}}_p)}^{-1}).$$ (4)

Also, m_F (z) is the solution of the following equation (Marcenko and Pastur 1967):

$$m_F(z)=\int \frac{dH(\tau )}{\tau (1-\gamma -\gamma z{m}_F(z))-z},$$ (5)

as is used in Theorem 1.

Theorem 1 Let λ > 0 be fixed, under H₀ and assumptions A1–A4, we have

$$\frac{\sqrt{p}(\frac{1}{p}RHT(\gamma )-{\mathrm{\Theta }}_1(\lambda ,\gamma ))}{{(2{\mathrm{\Theta }}_2(\lambda ,\gamma ))}^{1/2}}\Rightarrow N(0,1),\mathit{\text{where}}$$ (6)

$${\mathrm{\Theta }}_1(\lambda ,\gamma )=\frac{1-\lambda m_F(-\lambda )}{1-\gamma (1-\lambda m_F(-\lambda ))},\mathit{\text{and}}$$ (7)

$${\mathrm{\Theta }}_2(\lambda ,\gamma )=\frac{1-\lambda m_F(-\lambda )}{{(1-\gamma +\gamma \lambda m_F(-\lambda ))}^3}-\lambda \frac{m_F(-\lambda )-\lambda m_F^{\prime }(-\lambda )}{{(1-\gamma +\gamma \lambda m_F(-\lambda ))}^4}.$$ (8)

Remark 1 i) When γ < 1, (6) holds even with λ = 0, and in that case, we have Θ₁(0, γ) = 1/(1 − γ) and Θ₂(0, γ) = 1/(1 − γ)³, which gives the asymptotic null distribution of HT statistics as stated on page 314 of Bai and Saranadasa (1996).

ii) When Σ_p = I_p, m_F (−λ) has a simple analytical form, and the corresponding Θ₁(λ, γ) and Θ₂(λ, γ) can be written out explicitly (see equations (13) and (14) below).

In practice we estimate m_F (−λ), Θ₁(λ, γ) and Θ₂(λ, γ) by m_Fn,p(−λ), Θ̂_1,n(λ) and Θ̂_2,n(λ), respectively, where m_Fn,p(·) is given by (4), and Θ̂_j,n(λ), j = 1, 2 are obtained by substituting in (7) and (8), γ by p/n, m_F (−λ) by m_Fn,p(−λ) and $m_F^{\prime }(-\lambda )$ by $m_{F_{n,p}}^{\prime }(-\lambda )$ where $m_{F_{n,p}}^{\prime }(-\lambda )=\text{tr}({({\mathbf{S}}_n+\lambda {\mathbf{I}}_p)}^{-2})/p$. The following result proves the consistency of the estimators.

Proposition 1 Under the assumptions of Theorem 1, as n → ∞,

$$\sqrt{n}({\mathrm{\Theta ̂}}_{1,n}(\lambda )-{\mathrm{\Theta }}_1(\lambda ,\gamma ))\stackrel{P}{\to }0;$$ (9)

$$\mathit{\text{and}}{\mathrm{\Theta ̂}}_{2,n}(\lambda )-{\mathrm{\Theta }}_2(\lambda ,\gamma )\stackrel{P}{\to }0.$$ (10)

By virtue of Proposition 1, we have the following corollary of Theorem 1.

Corollary 1 Under the assumptions of Theorem 1, we have

$$\frac{\sqrt{p}(\frac{1}{p}RHT(\lambda )-{\mathrm{\Theta ̂}}_{1,n}(\lambda ))}{{(2{\mathrm{\Theta ̂}}_{2,n}(\lambda ))}^{1/2}}\Rightarrow N(0,1),$$ (11)

Observe that Corollary 1 gives a practical and efficient way of determining the asymptotic cut-offs for the RHT test. Note that, the quantities Θ̂_1,n(λ) and Θ̂_2,n(λ) are completely data-driven, and their computation does not require any knowledge of the true covariance matrix Σ beyond its positive definiteness.

The next two results characterize the asymptotic power of the proposed test.

Theorem 2 Suppose that λ > 0, assumptions A1–A4 hold, and the “effect size” ‖δ_n‖= cn^−θ for some positive constant c and θ ∈ (0, 1/4), where ${\mathit{\delta }}_n={\mathbf{\Sigma }}_p^{-1/2}{\mathit{\mu }}_p$ and μ_p is the true mean. Denote the power of RHT(λ) as β_n(δ_n) ≔ Pδ_n (RHT(λ) > ξ_p,γ,λ(1 − α)), where ξ_p,γ,λ(1 − α) is the (1 − α) quantile of the asymptotic distribution of RHT(λ) under H₀. Then, β_n(δ_n) → 1 as n → ∞.

Proposition 2 Suppose that λ > 0, Σ_p = I_p, assumption A2 holds, and n^1/4 ‖δ_n‖→ c > 0 as n → ∞. Then, the power of RHT(λ) satisfies

$${\beta }_n(\mathit{\delta })\to \mathrm{\Phi }(-{\xi }_{\alpha }+\frac{c^2{\mathrm{\Theta }}_1(\lambda ,\gamma )}{\sqrt{2\gamma {\mathrm{\Theta }}_2(\lambda ,\gamma )}}),\mathit{\text{as}}n\to \mathrm{\infty },$$ (12)

where Φ denotes the c.d.f. of N(0, 1),

$${\mathrm{\Theta }}_1(\lambda ,\gamma )=m_{F_{\gamma }}(-\lambda )=\frac{1}{2\gamma \lambda }[-(1-\gamma +\lambda )+\sqrt{{(1-\gamma +\lambda )}^2+4\gamma \lambda }],$$ (13)

$${\mathrm{\Theta }}_2(\lambda ,\gamma )=\frac{m_{F_{\gamma }}(-\lambda ){(1+\gamma m_{F_{\gamma }}(-\lambda ))}^2(1-\lambda m_{F_{\gamma }}(-\lambda ))}{1+\gamma \lambda {(m_{F_{\gamma }}(-\lambda ))}^2}.$$ (14)

To facilitate the flow of the paper, we defer the proofs of the above theoretical results to Section 5. Noted that the proof of Theorem 1 (specifically, Lemma 2 in Section 5) employs a novel technique following Ledoit and Péché (2010), based on the Stieltjes transform of the empirical spectral measure of a random matrix. This technique, useful in proving certain limit theorems in random matrix theory, essentially reduces the problem of deriving the asymptotic distribution of the RHT statistic to that of proving the asymptotic convergence of certain quadratic forms involving the sample covariance matrix. We deem this as a theoretical contribution of the paper.

Unlike the classical Hotelling’s T² test, the power function of the RHT test cannot be expressed in a simple form when λ > 0. As can be seen from the expansion (26) of the normalized RHT statistic in Section 5, when μ ≠ 0, the main contribution to the power of the RHT test comes from the quantity ${\mathit{\delta }}_n^T{\mathbf{\Lambda }}_n{\mathit{\delta }}_n$, where Λ_n is distributed as the inverse of W_n +λΣ⁻¹ and (n − 1)W_n has the p-dimensional Wishart distribution with n − 1 degrees of freedom. If the spectral norm of Σ and Σ⁻¹ are bounded above, the same is true for Λ_n and ${\mathbf{\Lambda }}_n^{-1}$. In that setting, the power of the RHT test primarily depends on the norm of the vector δ_n. The structure of δ_n shows that, if μ is aligned to the direction of the smallest eigenvectors of Σ, then the power will tend to be large; while if μ is aligned to the eigenvectors associated with the leading eigenvalues of Σ, then the power can deteriorate somewhat. However, unless the covariance matrix Σ is badly conditioned, the power will depend mainly on the l² norm of μ and not much on the structure of Σ. In the end, it is worth pointing out that, when γ ≥ 1, Theorem 2 and Proposition 2 do not hold for λ = 0 (see Simulation 1 in Section 5). This suggests that RHT(λ) for λ > 0 and HT (=RHT(0)) are intrinsically different, and the former is more suitable if p ≈ n or p ≥ n.

2.2 A testing procedure based on resampling

Proteomics data may have substantial missingness and small sample sizes, and thus the asymptotic null may not well approximate the empirical null for RHT statistic. In this section, we propose a non-parametric approach to estimate the p-values. The idea is to use Independent Null, which can be characterized through resampling, to approximate the Dependent Null (the true null). Specifically, we propose to select the tuning parameter λ through minimizing the difference between the asymptotic (1 − α)-quantile of the Independent Null and the asymptotic (1 − α)-quantile of the Dependent Null, where α is the target significance level; then based on the selected λ, the (1 − α) quantile of the empirical Independent Null from resampled data is used as the critical cutoff value. We describe the details of these steps below.

Step 1. Resampling. In the proteomics data, the proportion of missingness among proteins can be highly variable due to the dynamic nature of the mass spectrometer sampling, and different missingness patterns may affect the test statistics. Hence, we choose to preserve the same missing data pattern and proceed with the resampling procedure as follows. Denote the pool of non-missing measurements (concentration log ratios) of all proteins in all samples as Ω. Usually, Ω has been standardized to zero sample mean and unit sample standard deviation in the preprocessing normalization step. In each resampling iteration, we first randomly sample p × n numbers from Ω without replacement and record them in a matrix X̃_p×n. We preserve the same missing pattern in X̃ as the observed data by setting x̃_ij = NA for those {(i, j) : x_ij = NA}. In this resampling procedure, the log ratios within each random set are nearly independent, and hence $\widetilde{{\mathit{x}}_{\mathit{i}}}$ can be viewed as having zero mean and identity covariance matrix. We will refer to the null distribution of the RHT statistics based on the resampled data as Independent Null (IN), while the null distribution of the RHT statistic based on the observed data as Dependent Null (DN).

Step 2. Choosing λ. We use the asymptotic approximation to choose the degree of regularization (λ), such that the tail of the IN well approximates the tail of the DN. The former is then used to derive the critical value of the test statistics in Step 3. Specifically, for each λ, we first calculate (Θ̂_1,n(λ), Θ̂_2,n(λ)) based on the observed data X_p×n, and $({\widehat{\tilde{\mathrm{\Theta }}}}_{1,n}(\lambda );{\widehat{\tilde{\mathrm{\Theta }}}}_{2,n}(\lambda ))$ based on resampled data X̃_p×n. Then for a target significant level α, we calculate the difference between the asymptotic (1 − α) quantile of DN and IN:

$$D_{\alpha }(\lambda )=({\mathrm{\Theta ̂}}_{1,n}(\lambda )+\frac{\sqrt{2}{\xi }_{1-\alpha }}{\sqrt{p}}{\mathrm{\Theta ̂}}_{2,n}{(\lambda )}^{1/2})-({\widehat{\tilde{\mathrm{\Theta }}}}_{1,n}(\lambda )+\frac{\sqrt{2}{\xi }_{1-\alpha }}{\sqrt{p}}{\widehat{\tilde{\mathrm{\Theta }}}}_{2,n}{(\lambda )}^{1/2})$$ (15)

where ξ_1−α is the (1 − α) quantile of standard normal N(0, 1). We then select λ as:

$$\mathrm{\lambda ̂}=\text{min}(\mathrm{\Gamma })=\text{min}(\{\lambda :\lambda \in \mathrm{\Lambda },|D_{\alpha }(\lambda )|<\delta \}),$$ (16)

where Λ is a prespecified selecting range for λ, and δ is a small positive value. Here Γ is the collection of λ values, based on which the tail of IN is rather close to the tail of the DN. In Simulations and Application sections, we set Λ = [0.01, 10] and δ = 0.5 to ensure Γ ≠ ∅. We then choose to be conservative on the regularization level by selecting the minimum value of Γ, which further helps to control the type I error rate. Note that when λ = 0, the RHT statistic becomes the HT statistic, which is invariant to correlation structure in the data. Consequently, DN is stochastically the same as the IN when λ = 0, which suggests E(D_α(0)) = 0. On the other hand, when λ → ∞, RHT statistic does not make use of the correlation structure among X and becomes rather similar to the SAT statistic, which could have inflated type I error rate when correlation is present. In some sense, RHT statistic is between HT and SAT, aiming to achieve a better balance between type I error rate and power.

The calculation of D_α(λ) takes the eigenvalues of sample covariance matrix from IN and DN as input, where the latter comes from observed data. To obtain the eigenvalues from IN, we independently resample the observed pathway data 100 times, and derive the corresponding ${\{{\mathrm{\lambda ̂}}^{\tau }\}}_{\tau =1}^{100}$. The median of this set is selected to be the final ${\mathrm{\lambda ̂}}_{sel}=\text{median}{\{{\mathrm{\lambda ̂}}_{\tau }\}}_{\tau =1}^{100}$.

Step 3. Calculating p-value. Once λ̂_sel is chosen, the empirical distribution of the test statistics derived from the resampled data can be used to calculate the p-value. Particularly, let RHT(λ̂_sel) denote the test statistics calculated from the observed data X_p×n; and ${\{\mathrm{R}\mathrm{H}{\mathrm{T}}_0^j({\mathrm{\lambda ̂}}_{sel})\}}_{j=1}^M$ denote the test statistics calculated from M independent resampled data sets ${\{{\mathbf{X̃}}_{p\times n}^j\}}_{j=1}^M$. The p-value is then calculated as

$$\mathrm{p}-\text{value}=\frac{1}{M}\sum _{j}I({\mathrm{R}\mathrm{H}\mathrm{T}}_0^j({\mathrm{\lambda ̂}}_{sel})>\mathrm{R}\mathrm{H}\mathrm{T}({\mathrm{\lambda ̂}}_{sel})).$$ (17)

2.3 Modified RHT using Bootstrap

When γ = p/n value is large, the estimates Θ̂_1,n(λ) and Θ̂_2,n(λ) tend to have large variance, which sometimes hurts the power of the proposed test, especially when the signal size is small or moderate (Figure S3). To circumvent this problem, we introduce a modified ‘bootstrap-like’ statistic, RHT_Bootstrap, for large pathways. The idea is to first focus on small pathway subsets, for which signals can be efficiently captured by RHT statistics; and then average the information across all subsets. The detailed procedure is as follows:

1. Sample without replacement B subsets of variables, so that the b^th subset harbors p_b features;

2. Calculate RHT_b for each subset;

3. Report the average $\mathrm{R}\mathrm{H}{\mathrm{T}}_{\mathit{\text{Bootstrap}}}=\frac{1}{B}{\sum }_b\mathrm{R}\mathrm{H}{\mathrm{T}}_b$.

Here, if p_b is small, RHT_Bootstrap behaves similarly to SAT and may not take adequate account of the correlation among variables. Furthermore, in Theorem 2, we have shown that, unlike the HT test, the power of the RHT test converges to 1 even when p/n is close to 1. Therefore, choosing a p_b as large as possible while γ_b = p_b/n < 1 is expected to maintain good power, while guarding against inflated type I error rates and taking maximal advantage of correlation among variables. We used p_b = n−2 in Simulations and Application sections. Meanwhile, a convenient choice of λ for RHT_Bootstrap is $\frac{1}{B}{\sum }_b{\mathrm{\lambda ̂}}_b$, where λ̂_b is the selected tuning parameter for the b^th subset according to equation (16). The advantage of RHT_Bootstrap versus RHT is illustrated with simulation examples in Supplementary Information.

We summarize the procedures of the proposed test in the following two algorithms.

3 SIMULATIONS

We compare the performance of the proposed RHT and RHT_Bootstrap tests with several other alternatives: (1) HT test; (2) HT_Bootstrap (similar to RHT_Bootstrap); (3) SAT; (4) Maxmean statistic (Efron and Tibshirani 2007); (5) T_D test (Dempster 1958; Dempster 1960); (6) T_BS test (Bai and Saranadasa 1996); and (7) F⁺ test (Srivastava 2007). The HT test is only performed for p < n cases; while RHT_Bootstrap, HT_Bootstrap and F⁺ are only performed for p ≥ n cases. For RHT, RHT_Bootstrap(p > n), HT_Bootstrap(p > n), SAT, and Maxmean, we calculate the p-values using the resampling procedure described in Section 2.2. For HT, T_D, T_BS, and F⁺, we use their asymptotic distributions to obtain the corresponding significance levels (see Section 1 of Supplementary Information for details of these statistics and their asymptotic approximations). Note that with missingness, those asymptotic distributions can be far from the empirical null distributions, especially with limited sample size. We see from the following simulations that the tests based on these asymptotic distributions often cannot control type I error rate.

Algorithm 1.

A regularized Hotelling’s T² test

For a pathway consisting of p proteins measured on n samples, Xp×n, estimate the sample covariance matrix Sn and obtain the eigenvalues. Calculate λ̂sel, based on which the asymptotic (1 − α) quantiles of the RHT statistic of the resampled data is rather close to that of the observed data. Plug in λ̂sel and obtain the observed RHT statistic for the pathway of interest, RHT(λ̂sel). Resample observed log ratios M times, preserving the missing data pattern of X, and generate M null pathways ${\mathbf{X}}_1^0,\dots ,{\mathbf{X}}_M^0$. For each ${\mathbf{X}}_j^0$, j = 1, …, M, compute $\mathrm{R}\mathrm{H}{\mathrm{T}}_j^0({\mathrm{\lambda ̂}}_{sel})$. Calculate p-value by comparing RHT(λ̂sel) with ${\{\mathrm{R}\mathrm{H}{\mathrm{T}}_j^0({\mathrm{\lambda ̂}}_{sel})\}}_j$.

Algorithm 2.

A regularized Hotelling’s T² test with Bootstrap

For a pathway consisting of p proteins measured on n samples with p > n − 1, we independently sample B subsets, each contains n − 2 proteins. For each subset Xb, b = 1, …, B, compute ${\mathrm{\lambda ̂}}_{sel}^b$; then set ${\mathrm{\lambda ̂}}_{sel}=\frac{1}{B}{\sum }_b{\mathrm{\lambda ̂}}_{sel}^b$. Calculate the observed pathway statistic as $\mathrm{R}\mathrm{H}{\mathrm{T}}_{\mathit{\text{Bootstrap}}}({\mathrm{\lambda ̂}}_{sel})=\frac{1}{B}{\sum }_b\mathrm{R}\mathrm{H}{\mathrm{T}}_b({\mathrm{\lambda ̂}}_{sel})$. Resample observed log ratios M times, preserving the missing data pattern of X, and generate M null pathways ${\mathbf{X}}_1^0,\dots ,{\mathbf{X}}_M^0$. For each ${\mathbf{X}}_j^0$, j = 1, …, M, make B subsets of proteins as outlined in Step 1, and calculate $\mathrm{R}\mathrm{H}{\mathrm{T}}_{\mathit{\text{Bootstrap}},j}^0({\mathrm{\lambda ̂}}_{sel})$ as in Step 3. Calculate p-value by comparing RHTBootstrap(λ̂sel) with ${\{\mathrm{R}\mathrm{H}{\mathrm{T}}_{\mathit{\text{Bootstrap}},j}^0({\mathrm{\lambda ̂}}_{sel})\}}_j$.

Simulation 1: Impact of sample size when p < n and p/n → 1.

In this simulation, we compare the performance of the proposed RHT test with that of the HT test under p/n → 1 scenario. Specifically, for each n ∈ {5, 10, 15, …, 100}, we set p = n−2 and simulate X ~ N(μ, Σ), where μ = (0.5, …, 0.5)^T, σ_ii = 1 and σ_ij = 0.2 (i ≠ j). The type I error rate and power of the two tests are calculated based on 1000 simulations. The results are illustrated in Figure 2. HT test controls the type I error rate properly, while the proposed RHT test tends to be slightly conservative. When power is considered, HT test performs extremely poorly, which is consistent with the theoretical result in Bai and Saranadasa (1996), while the power of RHT test increases to 1 with increasing sample size, consistent with our Theorem 2 in Section 2.1. This suggests that the regularization employed in RHT statistics is much preferred for handling data with p close to n.

Figure 2.

Impact of sample size when p = n − 1. (a): Type I error. The horizontal black line indicates the targeted significance level of 0.05. (b): Power at alternative μ = (0.5,‖,0.5)^T.

Simulation 2: Impact of correlation and missingness when p < n.

In this simulation, we consider a traditional setting with p < n (p = 20 and n = 50), and evaluate the impact of correlation and missingness on the performances of various tests. Specifically, we simulate X ~ N(μ, Σ), with μ = 0 under the null and {μ₁ = … = μ₁₀ = 0, μ₁₁ = … = μ₁₅ = 0.2, μ₁₆ = … = μ₂₀ = −0.2} under the alternative; σ_ii = 1 and σ_ij = ρ for i ≠ j. We consider two different correlation settings: completely independent (ρ = 0) and moderately pair-wise correlated (ρ = 0.3). For each ρ, we generate data in three ways: (1) without missingness; (2) with 20% missing completely at random (MCAR) (in our proteomics data, the missingness rate is about 25%); and (3) with 20% MCAR and apply a k-nearest neighbors (KNN) imputation method (Troyanskaya et al. 2001) to impute the missing values. This results in six different settings.

We simulate 1000 independent data sets for each setting, and evaluate the type I error rate and power of the competing tests at 0.05 p-value cutoff. The results are illustrated in Figure 3 (a) and (c). SAT and Maxmean could induce inflated type I error rates when correlations are present. T_BS and HT tests fail to control type I error rates when missingness is not imputed. Only RHT and T_D manage to control the type I error rates under all settings. In fact, with missingness, the asymptotic null distributions for HT, T_BS and T_D can substantially deviate from the corresponding empirical null distributions, resulting inaccurate significance level approximations. The deviation can depend on the pattern and extent of missing data. Concerning power, when correlations are present, RHT has better power than T_D. In addition, for both ρ = 0 and ρ = 0.3, the presence of missingness decreases the power of both RHT and T_D. On the other hand, regardless of missing data, the presence of correlation slightly increases the power of RHT. This suggests that making use of correlations among variables in a proper way may enhance the power of pathway analyses.

Figure 3.

Comparing type I error rates (a),(b) and power (c),(d) of competing tests at 0.05 p-value cutoff. (a) and (c) are for Simulation 2: p = 20, n = 50. (b) and (d) are for Simulation 3: p = 20, n = 10. In both Simulation 2 and 3, the correlation matrix Σ has diagonal elements: σ_ii = 1, and off diagonal elements: σ_ij = ρ (i, j = 1,…,p).

Simulation 3: Impact of correlation and missingness when p > n.

In this simulation, we set p = 20 and n = 10 to mimic the dimensionality of our real application. We simulate the data in the same way as described in Simulation 2. In addition, here, we evaluate three additional tests, F⁺, RHT_Bootstrap and HT_Bootstrap. Note the parametric HT test is ill-positioned when p > n and is not compared. The results are shown in Figure 3 (b) and (d). RHT, RHT_Bootstrap, HT_Bootstrap and T_D evidently control type I error rates. Among those, RHT_Bootstrap is the most powerful under all settings. The presence of missingness dramatically decreases the power of T_D, but has much less effect on the power of RHT and RHT_Bootstrap.

In both Simulation 2 and 3, the performances of RHT test (or RHT_Bootstrap) with and without imputation are similar. Imputation helps HT and F⁺ tests to control type I error rate when there is missingness in the data, and imputation increases the power of T_D test. However, even with imputation, when there is correlation, the performance of RHT test when p < n or RHT_Bootstrap when p > n remains the best.

In Supplementary Information, we compare the performance of RHT test with and without bootstrap on simulated examples, and demonstrate that the former has superior power than the latter when γ = p/n is large. Thus, to avoid confusion, for p > n scenarios, we will only consider RHT_Bootstrap statistic in the following simulations.

Simulation 4: Impact of complicated correlation structure.

We also perform simulations using more complicated correlation structures. Specifically, ${\mathbf{\Sigma }}_1^{\prime }$ is defined as a diagonal matrix with ${\sigma }_{ii}^{\prime }\propto i(i=1,\cdots ,p)$. For ${\mathbf{\Sigma }}_2^{\prime }$, the off diagonal elements are set to be ${\sigma }_{ij}^{\prime }={\rho }_{ij}^{\prime }\sqrt{{\sigma }_{ii}^{\prime }{\sigma }_{jj}^{\prime }}$, where ${\rho }_{ij}^{\prime }~0.5\delta (0)+0.5N(0.5,{0.1}^2)$; and the diagonal elements are set to be the same as ${\mathbf{\Sigma }}_1^{\prime }$. In addition, we define ${\mathbf{\Sigma }}_3^{\prime }$ with off diagonal elements ${\sigma }_{ij}^{\prime }={0.8}^{|i-j|}\sqrt{{\sigma }_{ii}^{\prime }{\sigma }_{jj}^{\prime }}$ and the same diagonal elements as ${\mathbf{\Sigma }}_1^{\prime }$.

We set p = 20 and consider two different sample sizes n = 50 and 10. We generate missingness completely at random as in Simulation 2 and 3, and impute the missing values using KNN imputation. The results, which are shown in Figure 4, lead to the same conclusion that RHT and RHT_Bootstrap perform the best among the competing tests in terms of type I error rate and power. This suggests that the proposed test works well even when the variables in the pathway have somewhat different variances. Note that under complex correlation patterns, correlation does not necessarily increase type I error rates. Ignoring correlation structure may also result in overall reduction in both type I error rate and power.

Figure 4.

Comparing type I error rates (a),(b) and power (c),(d) of the competing tests at 0.05 p-value cutoff, with complex correlation patterns. For (a) and (c), p = 20, n = 50. For (b) and (d), p = 20, n = 10. For the three types of correlations, ${\sigma }_{ii}^{\prime }\propto i(i=1,\cdots ,p)$. For ${\mathbf{\Sigma }}_1^{\prime },{\sigma }_{ij}^{\prime }=0$. For ${\mathbf{\Sigma }}_2^{\prime },{\rho }_{ij}^{\prime }~0\cup N(0.5,{0.1}^2)$, and ${\sigma }_{ij}^{\prime }={\rho }_{ij}^{\prime }\sqrt{{\sigma }_{ii}^{\prime }{\sigma }_{jj}^{\prime }}$. For ${\mathbf{\Sigma }}_3^{\prime },{\sigma }_{ij}^{\prime }={0.8}^{|i-j|}\sqrt{{\sigma }_{ii}^{\prime }{\sigma }_{jj}^{\prime }}$.

Simulation 5: Impact of missing not completely at random and imputation.

We further simulate data with not-missing-completely-at-random (not-MCAR) and compare competing tests without and with imputation. The missing probabilities, which range from 20% to 40%, are simulated to be positively correlated with the absolute values of the ratios, such that extreme ratios have higher probabilities to be missing. Figure 5 shows the results of competing tests on not-MCAR data with possible correlations for p < n (p = 20, n = 50) and p > n (p = 20, n = 10). With not-MCAR data, SAT, Maxmean, T_BS, HT and F⁺ tests still could not control type I error rates. KNN-imputation could potentially further increase the type I error rates for SAT, Maxmean, HT and F⁺. On the other hand, due to the conservative design of RHT test, even when data are not-MCAR, the proposed RHT_Bootstrap test could still control the type I error rates within a reasonable range, and it is substantially more powerful than T_D or HT_Bootstrap tests.

Figure 5.

Comparing type I error rates (a),(b) and power (c),(d) of competing tests at 0.05 p-value cutoff, with not-missing-completely-at-random (NMCAR). For (a) and (c), p = 20, n = 50. For (b) and (d), p = 20, n = 10. In both simulations, the correlation matrix Σ has diagonal elements: σ_ii = 1, and off diagonal elements: σ_ij = ρ (i, j = 1,…,p).

In summary, SAT test could not control type I error rate when correlation is present. Maxmean behaves similarly as or worse than SAT. HT and HT_Bootstrap have very poor performance when p is close to or greater than n, due to the intrinsic limitation of HT statistics. The other few tests, T_D, T_BS, and F⁺ are all parametric tests and are based on asymptotic theories derived under non-missing settings. However, when missingness is present, the distribution of these test statistics vary a lot with the degree and pattern of missingness. Moreover, the calculation of T_D, T_BS, and F⁺ statistics involves the estimation of S², which could be highly non-robust when missing data is present. Furthermore, the sample size in our problem is relatively small. Thus, the asymptotic distributional approximations to these parametric tests do not hold well, and then, not surprisingly, their performances are less competitive. In contrast, for RHT, we employ a resampling-based testing procedure, in which the pattern of missingness is explicitly accounted for, and the cut-off values are obtained empirically, as opposed to relying entirely on the asymptotic theory for the complete data scenario. Thus, RHT performs more favorably with MCAR data. We recommend using RHT test when p < n and RHT_Bootstrap for p > n. Also, the proposed resampling test strategy will not apply to T_D, T_BS, or F⁺. This is because these statistics are not invariant to Σ, while the resampling can approximate only the null distribution under Σ = I. In contrast, for RHT, by changing the regularization parameter λ, we are able to identify a resampling distribution that well approximates the tail of the true null distribution under the same regularization level, and hence properly control the type-I error rate of the test.

We also perform simulations for two-sample testing and get similar conclusions (see Supplementary Information).

4 APPLICATION

Postmenopausal hormones have been widely used in the United States and elsewhere, for the control of vasomotor and other menopausal symptoms. Following some years of observational epidemiologic study, the health benefits and risks of two widely used hormone therapy regimens were more formally tested during 1993–2005 in randomized controlled trials among more than 27,000 healthy U.S. postmenopausal women, as a part of the Women’s Health Initiative (WHI). The estrogen plus progestin regimen studied in a trial of 16,608 postmenopausal women with uterus stopped early in 2002 when health risks were judged to exceed benefits (Rossouw et al. 2002). The risks included elevations in breast cancer, stroke, coronary heart disease, and venous thromboembolism, while fracture reduction was a noteworthy benefit. The estrogen-alone trial among 10,739 women who were post-hysterectomy was stopped early in 2004 (Anderson et al. 2004), primarily because of a stroke elevation of similar magnitude to that seen for estrogen plus progestin. See Prentice and Anderson (2007) for further details. A nested case-control study, involving candidate blood biomarkers of inflammation, coagulation, and lipids, was carried out in an attempt to elucidate cardiovascular aspects of these findings, but failed to identify important mediators of the observed clinical effects (Kooperberg et al. 2007; Rossouw et al. 2008).

Proteomic discovery work was initiated soon thereafter. The application considered here is one component of this proteomic initiative. This component sought circulating proteins, and protein pathways, for which the concentrations changed following the initiation of hormone therapy. Specifically, this proteomic discovery project involved the comparison of serum protein concentrations one year after randomization to corresponding pre-randomization concentrations for 100 women who adhered to their assigned active regimens during the first year of hormone therapy trial participation. The research agenda for the overall initiative also involved proteomic case-control comparisons for coronary heart disease, stroke, and breast cancer using the same proteomic platform. Novel proteins, and protein pathways, that changed concentration following hormone therapy, and that were associated with the risk of study diseases, are being studied further as potential mediators of hormone therapy clinical effects. This ongoing research involves assessing the concentrations of proteins of interest in individual cases and controls in the hormone therapy trial cohorts. Some results from this research strategy have recently been reported (Prentice et al. 2010).

The 100 women in this proteomic study component included 50 who were randomly selected from among women assigned to active estrogen plus progestin and adhered (80% or more pills taken) to their assigned treatment during the first year following randomization, and a corresponding 50 assigned to active estrogen-alone. An Intact Protein Analysis System (IPAS) platform (Faca et al. 2006) was used in these studies. The 1-year and baseline serum specimens were labeled with either a heavy or a light acrylamide, and combined. Because the IPAS measurement process is rather time-consuming and expensive, serum samples were pooled to form 10 specimen pool pairs. Each pool is formed from an equal volume of serum from 10 women, resulting 5 pool pairs for estrogen plus progestin and 5 for estrogen-alone (Pitteri et al. 2009). Serum samples were obtained from fasting blood samples according to a standard protocol, and stored at −80 degree until needed for analysis. Baseline and 1-year pool pairs, from randomly selected sets of 10 women from the regimen-specific sets, were formed on the day needed for mass spectrometry analyses. Peptide concentration ratios were extracted as the ratio of areas under mass spectrum peaks separated by the mass difference between heavy and light acrylamide, and protein concentration ratios were estimated by reconstructing proteins from their peptide signatures. Missing data may arise if mass spectrum peaks are not clearly above background, or if quantified peptides do not identify the protein from which they derive with certainty.

The result of estimated protein concentration ratios can be displayed as a matrix X_p×n, where n = 10 is the number of pool pairs, and p = 369 is the total number of proteins having relative concentration ratios estimated, and the (i, j) element of the matrix is the log2 ratio of protein concentration at 1-year compared to baseline for the i-th protein in the j-th serum pool pair.

The probability of missingness for a given protein may depend on abundance level in the generating pools, on the protein sequence (e.g., acrylamide binds to cysteine residues, so only proteins containing such residues will have concentration ratio estimated) and protein mass. However, missingness rates for proteins quantified is not expected to depend on the value of the 1-year to baseline concentration ratio; nor is it expected to depend on the concentration ratios for other quantified proteins. Hence, a missing completely at random assumption seems plausible in this context. We note that the MCAR assumption may not be satisfied for some proteomics studies, especially when protein abundance level instead of ratio is measured for each protein. It would be beneficial to check the MCAR assumption before applying the proposed method. For this application, Figure S5 in Supplementary Information shows the boxplots of pooled absolute protein log2 ratios stratified by # missing observations, for the 369 quantified proteins. We do not observe a clear trend between the absolute log 2 ratios and missingness rate. Figure 6(f) also provides a visualization of the missing patterns of a few example pathways and of all the proteins in the data set (roughly 25% of the data are missing).

Figure 6.

Heatmaps of protein log concentration ratios from the five significant pathways and from all proteins in the data set. Each row represents the ratios from a protein and each column represents one IPAS experiment. Missing values are displayed in white. Positive, zero and negative log concentration ratios are shown in red, black, and green. (a)–(e) are the heatmaps of complement and coagulation cascades, cell communication, ECM receptor interaction, focal adhesion and TGF-beta signaling pathways, respectively. (f) is the heatmap of all proteins in the data.

We apply the proposed RHT test to 18 KEGG human disease pathways (Kanehisa 2002) having 5 or more proteins quantified in our dataset. Our goal is to detect whether any of the pathways show concerted differential abundance between baseline and 1-year. Before we perform the test of μ = 0 for the log ratios, we first scale the log ratios for each protein by its estimated standard deviation plus a constant (= 0.17, which is the 10% quantile of all estimated standard deviations). For pathways having fewer than 9 proteins, we use RHT statistics; while for pathways having 9 or more proteins, we use RHT_Bootstrap statistics with $B=[(\frac{5p}{n-1})]$. P-values were calculated based on 10,000 resamplings. Considering that pathways might have overlapping proteins and are not completely independent, Bonferroni correction for controlling 5% family-wise error rate (FWER) might be too conservative. Thus we choose a relatively loose FWER, 10%, which corresponds to a stringent 1% false discovery rate (FDR) (Benjamini and Hochberg 1995). At this cutoff, there are five significant pathways: complement and coagulation cascades, cell to cell communication, extracellular matrix (ECM) receptor interaction, focal adhesion and TGF-beta signaling pathways (see Table 1 for details). Figure 6(a)–(e) display the heatmaps of these five pathways. For comparison, the heatmap of all proteins in the data is shown in Figure 6(f). From the heatmaps we can see that, the five pathways we identified each has multiple proteins showing consistent patterns of red or green across experiments, corresponding to evidence of increased or decreased concentration after randomization. Specifically, by testing each protein in the selected pathway individually, we find that about 28.6%–60.0% proteins in each selected pathway have nominal p-value<0.05, suggesting the significant pathways are not wholly due to one or two proteins.

Table 1.

Significant KEGG pathways at FDR 1% and FWER of 10% (p-value cutoff = 0.1/18= 0.0056 by Bonferroni correction), identified from a postmenopausal hormone therapy proteomic study of the Women Health Initiative. P-values were calculated based on 10,000 resamplings.

Pathway names	# non-missing protein	P-value of RHT	P-value of SAT	P-value of HT/GHT	P-value of TD	P-value of TBS	% significant proteins*
Complement and coagulation cascades	45	0	0	0	0.002	0	22.2
Cell communication	12	0	0.001	0	0	0	16.7
Extracellular matrix receptor interaction	12	0	0	0	0	0	16.7
Focal adhesion	15	0	0	0	0	0	20.0
TGF-beta signaling	7	0.005	0.003	0.057	0.072	0	0.0

^*% of proteins with significant nominal p-value at 0.10 FWER (protein p-value cutoff of 0.10/369).

Related to this, one can ask whether these pathways may have otherwise emerged as hormone therapy-related by virtue of several individual proteins having concentration changes that meet multiple testing criteria. This is likely the case for the complement and coagulation cascade pathway, which includes multiple proteins (22.2%) showing strong evidence of differential abundance, i.e., with nominal protein p-value significant at 0.10 FWER (p-value below 0.1/369). In comparison, 11.4% of the proteins in the data passed this nominal p-value cutoff. The next three pathways, extracellular receptor, cell communication and focal adhesion pathways, which are substantially overlapping with 8 common proteins, each includes a higher than random percentage of proteins with nominal p-value significant at FWER 10%. These three pathways are closely related proceses with possible co-functioning (see Supplementary Figure S6 from KEGG website). On the other hand, besides the 8 common proteins, each pathway also has some unique proteins with suggestive significant p-values. For example, the focal adhesion pathway has the protein IGF1 with nominal p-value of 4.4e-05. The pathway TGF-beta signaling does not include any proteins meeting the FWER criterion (0.1/369). In fact, this pathway appears to be a novel and important finding that derives from correlated effects on several pathway proteins. The TGF-beta signaling pathway is commonly altered in human cancers (Pasche 2001). The pathway signals through the TGF-beta serine or threonine kinase receptors and downstream intercellular proteins of the SMAD transcription factor family to inhibit cell proliferation and induce apoptosis; the pathway also induces tumor progression via cell differentiation, migration, and adhesion. Much of the significance of the TGF-beta pathway derives (Figure 6(e)) from deviators in the beta E chain (INHBE) or the beta C chain (INHBC) of inhibin. Inhibins inhibit follicle stimulating hormone secretion; and inhibin secretion is enhanced by insulin-like growth factor 1, which is also elevated by post menopausal hormone therapy (Pitteri et al. 2009). Though speculative at this time, the observed effects on TGF-beta superfamily signaling could help to provide a unifying theme toward understanding hormone therapy effects on breast cancer, and possibly on observed hormone therapy effects more generally. The list of proteins in each of the five significant pathways is given in Supplementary Information.

For comparison, we also applied SAT, HT/F⁺, T_D and T_BS tests to this data set. Table S1 in Supplementary Information shows the number of significant pathways at FWER of 0.05 and 0.10 by Bonferroni correction for all the five tests including RHT. We can see that SAT, HT/F⁺ and T_BS tests each yields many ‘significant’ pathways. Since we have shown in simulations that those four tests often cannot control type I error rate under missingness and correlation, it is reasonable to suspect that many of the ‘significant’ pathways based on these tests are false positives. Among the 6 pathways identified by T_D test at FWER of 0.10, four overlap with the results of RHT test. However, T_D failed to identify the TGF-beta signaling pathway, and the other two pathways T_D identified, cell adhesion molecules and regulation of actin cytoskeleton pathways, seem less interpretable.

5 PROOFS

In this section, we prove the two theorems using a series of Lemmas. The proofs of the Lemmas are then outlined in the subsections while more technical details are provided in Supplementary Information. However, due to space limitations, we omit the proofs of Proposition 1, 2 and Corollary 1, which are either straightforward extensions of literature results or rather similar to the proofs of the two theorems.

5.1 Proof of Theorem 1

Under the null, $\sqrt{n}{\mathbf{X̅}}_n~N_p(\mathbf{0},{\mathbf{\Sigma }}_p)$ and is independent of S_n. Let $\mathbf{Z}=\sqrt{n}{\mathbf{\Sigma }}_p^{-1/2}{\mathbf{X̅}}_n$, then Z ~ N_p(0, I_p) and is independent of S_n. It follows $RHT(\lambda )={\mathbf{Z}}^T{\mathbf{\Sigma }}_p^{1/2}{({\mathbf{S}}_n+\lambda {\mathbf{I}}_p)}^{-1}{\mathbf{\Sigma }}_p^{1/2}\mathbf{Z}$. With Lemma 1 and 2, Theorem 1 follows immediately. The proofs of the following Lemmas are provided in sections 5.4–5.5.

Lemma 1 Under the same condition as Theorem 1, we have

$$\frac{\sqrt{p}(\frac{1}{p}RHT(\lambda )-\frac{1}{p}\mathit{\text{tr}}({({\mathbf{S}}_n+\lambda {\mathbf{I}}_p)}^{-1}{\mathbf{\Sigma }}_p))}{{(\frac{2}{p}\mathit{\text{tr}}({({\mathbf{S}}_n+\lambda {\mathbf{I}}_p)}^{-1}{\mathbf{\Sigma }}_p{({\mathbf{S}}_n+\lambda {\mathbf{I}}_p)}^{-1}{\mathbf{\Sigma }}_p))}^{1/2}}\Rightarrow N(0,1).$$ (18)

Lemma 2 Under the same condition as Theorem 1, we have

$$\sqrt{p}|\frac{1}{p}\mathit{\text{tr}}({({\mathbf{S}}_n+\lambda {\mathbf{I}}_p)}^{-1}{\mathbf{\Sigma }}_p)-{\mathrm{\Theta }}_1(\lambda ,\gamma )|\stackrel{P}{\to }0$$ (19)

$$\mathit{\text{and}}\frac{1}{p}\mathit{\text{tr}}({({\mathbf{S}}_n+\lambda {\mathbf{I}}_p)}^{-1}{\mathbf{\Sigma }}_p{({\mathbf{S}}_n+\lambda {\mathbf{I}}_p)}^{-1}{\mathbf{\Sigma }}_p)\stackrel{P}{\to }{\mathrm{\Theta }}_2(\lambda ,\gamma ),$$ (20)

5.2 Proof of Theorem 2

Let Z ~ N(0, I_p) and be independent of S_n. Under the alternative,

$$RHT(\lambda )=n{\mathbf{X̅}}_n^T{({\mathbf{S}}_n+\lambda {\mathbf{I}}_p)}^{-1}{\mathbf{X̅}}_n={(\mathbf{Z}+\sqrt{n}{\mathit{\delta }}_n)}^T{\mathbf{\Sigma }}_p^{1/2}{({\mathbf{S}}_n+\lambda {\mathbf{I}}_p)}^{-1}{\mathbf{\Sigma }}_p^{1/2}(\mathbf{Z}+\sqrt{n}{\mathit{\delta }}_n)={\mathbf{Z}}^T{\mathbf{\Sigma }}_p^{1/2}{({\mathbf{S}}_n+\lambda {\mathbf{I}}_p)}^{-1}{\mathbf{\Sigma }}_p^{1/2}\mathbf{Z}+2\sqrt{n}{\mathbf{Z}}^T{\mathbf{\Sigma }}_p^{1/2}{({\mathbf{S}}_n+\lambda {\mathbf{I}}_p)}^{-1}{\mathbf{\Sigma }}_p^{1/2}{\mathit{\delta }}_n+n{\mathit{\delta }}_n^T{\mathbf{\Sigma }}_p^{1/2}{({\mathbf{S}}_n+\lambda {\mathbf{I}}_p)}^{-1}{\mathbf{\Sigma }}_p^{1/2}{\mathit{\delta }}_n$$ (21)

Let ${\mathbf{W}}_n={\mathbf{\Sigma }}_p^{-1/2}{\mathbf{S}}_n{\mathbf{\Sigma }}_p^{-1/2}$. Then (n − 1)W_n, which is independent of Z, has a Wishart distribution with dimension p and degrees of freedom n − 1. Further define

$${\mathbf{\Lambda }}_n={\mathbf{\Sigma }}_p^{1/2}{({\mathbf{S}}_n+\lambda {\mathbf{I}}_p)}^{-1}{\mathbf{\Sigma }}_p^{1/2}={({\mathbf{W}}_n+\lambda {\mathbf{\Sigma }}_p^{-1})}^{-1}.$$

Consider the eigen-decomposition of Λ_n = Q^TLQ, where Q is an orthogonal matrix and L is a diagonal matrix with positive diagonal elements ζ_{n, 1} ≥ ζ_{n, 2} ≥ … ≥ ζ_{n, p}.

Lemma 3 Under assumptions A1–A4, {ζ_{n, i}} are bounded away from both zero and infinity asymptotically: ∃ $h_1^{\prime }\in (0,h_1]$ and $h_2^{\prime }<\infty$ satisfying

$$\underset{n\to \infty }{\text{lim inf}}{\zeta }_{n,p}\ge \frac{1}{{(1+\gamma )}^2+\lambda /h_1^{\prime }}a.s.$$ (22)

$$\mathit{\text{and}}\underset{n\to \infty }{\text{lim sup}}{\zeta }_{n,1}\le h_2^{\prime }/\lambda a.s.$$ (23)

Based on Lemma 3, we have

$$\sqrt{n}{\mathit{\delta }}_n^T{\mathbf{\Sigma }}_p^{1/2}{({\mathbf{S}}_n+\lambda {\mathbf{I}}_p)}^{-1}{\mathbf{\Sigma }}_p^{1/2}{\mathit{\delta }}_n=\sqrt{n}{\mathit{\delta }}_n^T{\mathbf{\Lambda }}_n{\mathit{\delta }}_n\ge \sqrt{n}{\zeta }_{n,p}{\Vert {\mathit{\delta }}_n\Vert }^2\stackrel{P}{\to }\infty ,$$ (24)

since $\sqrt{n}{\Vert {\mathit{\delta }}_n\Vert }^2\to \infty$ as n → ∞. Next, conditionally on W_n, ${\mathbf{Z}}^T{\mathbf{\Sigma }}_p^{1/2}{({\mathbf{S}}_n+\lambda {\mathbf{I}}_p)}^{-1}{\mathbf{\Sigma }}_p^{1/2}{\mathit{\delta }}_n={\mathbf{Z}}^T{({\mathbf{W}}_n+\lambda {\mathbf{\Sigma }}_p^{-1})}^{-1}{\mathit{\delta }}_n~N(0,{\mathit{\delta }}_n^T{\mathbf{\Lambda }}_n^2{\mathit{\delta }}_n)$. Then according to (23), it is easy to see ${\mathit{\delta }}_n^T{\mathbf{\Lambda }}_n^2{\mathit{\delta }}_n\le {\zeta }_{n,1}^2{\Vert {\mathit{\delta }}_n\Vert }^2\stackrel{P}{\to }0$. It follows that

$${\mathbf{Z}}^T{\mathbf{\Sigma }}_p^{1/2}{({\mathbf{S}}_n+\lambda {\mathbf{I}}_p)}^{-1}{\mathbf{\Sigma }}_p^{1/2}{\mathit{\delta }}_n\stackrel{P}{\to }0.$$ (25)

Then, using (21), we have the expansion

$$\frac{RHT(\lambda )-p{\mathrm{\Theta }}_1(\lambda ,\gamma )}{\sqrt{2p{\mathrm{\Theta }}_2(\lambda ,\gamma )}}=\frac{{\mathbf{Z}}^T{\mathbf{\Lambda }}_n\mathbf{Z}-p{\mathrm{\Theta }}_1(\lambda ,\gamma )}{\sqrt{2p{\mathrm{\Theta }}_2(\lambda ,\gamma )}}+\sqrt{\frac{n}{p}}\frac{1}{\sqrt{2{\mathrm{\Theta }}_2(\lambda ,\gamma )}}{\mathbf{Z}}^T{\mathbf{\Lambda }}_n{\mathit{\delta }}_n+\sqrt{\frac{n}{p}}\frac{1}{\sqrt{2{\mathrm{\Theta }}_2(\lambda ,\gamma )}}\sqrt{n}{\mathit{\delta }}_n^T{\mathbf{\Lambda }}_n{\mathit{\delta }}_n.$$ (26)

The first term on the right hand side of (26) converges in distribution to N(0, 1) by Theorem 1, the second term converges to 0 in probability by (25), and the third term diverges to infinity in probability by (24). Therefore, Theorem 2 follows.

5.3 Proof of Lemma 3

We first prove Lemma 3, which will then be used to prove Lemma 1. Denote the eigenvalues of W_n as η_n,1 ≥ η_n,2 ≥ ⋯ ≥ η_n,p. First,

$$1/{\zeta }_{n,p}={\text{max}}_{\mathbf{v}:\Vert \mathbf{v}\Vert =1}{\mathbf{v}}^T{\mathbf{\Lambda }}_n^{-1}\mathbf{v}\le {\text{max}}_{\mathbf{v}:\Vert \mathbf{v}\Vert =1}{\mathbf{v}}^T{\mathbf{W}}_n\mathbf{v}+\lambda {\text{max}}_{\mathbf{v}:\Vert \mathbf{v}\Vert =1}{\mathbf{v}}^T{\mathbf{\Sigma }}_p^{-1}\mathbf{v}={\eta }_{n,1}+\lambda /{\tau }_{p,p}.$$

It follows that ζ_n,p ≥ 1/(η_n,1 + λ/τ_p,p). In addition, since {τ_j,p} are bounded away from zero and infinity, and ${\eta }_{n,1}\stackrel{a.s.}{\to }{(1+\gamma )}^2$ as n → ∞ (Geman 1980), we shall have ${\text{lim inf}}_{n\to \infty }{\zeta }_{n,p}\ge \frac{1}{{(1+\gamma )}^2+\lambda /h_1^{\prime }}$ a.s. for some $h_1^{\prime }\in (0,h_1]$. On the other hand,

$$1/{\zeta }_{n,1}={\text{min}}_{\mathbf{v}:\Vert \mathbf{v}\Vert =1}{\mathbf{v}}^T{\mathbf{\Lambda }}_n^{-1}\mathbf{v}\ge \lambda {\text{max}}_{\mathbf{v}:\Vert \mathbf{v}\Vert =1}{\mathbf{v}}^T{\mathbf{\Sigma }}_p^{-1}\mathbf{v}=\lambda /{\tau }_{1,p}.$$

It follows that ${\text{lim sup}}_{n\to \infty }{\zeta }_{n,1}\le h_2^{\prime }/\lambda$ a.s. for some $h_2^{\prime }<\infty$.

5.4 Proof of Lemma 1

Recall that ${\mathbf{\Lambda }}_n={\mathbf{\Sigma }}_p^{1/2}{({\mathbf{S}}_n+\lambda {\mathbf{I}}_p)}^{-1}{\mathbf{\Sigma }}_p^{1/2}$. Decompose Λ_n as Q^TLQ, where Q is an orthogonal matrix and L is a diagonal matrix with positive diagonal elements ζ_n,1 ≥ ζ_n,2 ≥ … ≥ ζ_n,p. Then we have under the null, $\mathrm{R}\mathrm{H}{\mathrm{T}}_n(\lambda )={\mathbf{Z̃}}^T\mathbf{L}\mathbf{Z̃}={\sum }_{i=1}^p{\zeta }_{n,i}{{\mathrm{z̃}}_i}^2={\sum }_{i=1}^p{\zeta }_{n,i}{w}_{n,i}$, where ${\{w_{n,i}\}}_{i=1}^n$ are i.i.d. χ² random variables with 1 degree of freedom, and Z̃ = QZ.

Let T_p,i = ζ_n,i(w_n,i−1), $S_p={\sum }_{i=1}^p{T}_{n,i}$, and ${\vartheta }_p={(2{\sum }_{i=1}^p{\zeta }_{n,i}^2)}^{1/2}$. It is easy to see that $\sqrt{2p}{\zeta }_{n,n}\le {\vartheta }_p\le \sqrt{2p}{\zeta }_{n,1}$. Conditional on Θ = {ζ_n,i}_n,i, we have $E(T_{p,i}|\mathbf{\Theta })=0,E(T_{p,i}^2|\mathbf{\Theta })=2{\zeta }_{n,i}^2$, and ${\sum }_{i=1}^{p}E({(\frac{T_{p,i}}{{\vartheta }_p})}^2|\mathbf{\Theta })=1$. It follows that

$${\sum }_{i=1}^{p}E({\left|\frac{T_{p,i}}{{\vartheta }_p}\right|}^2;|\frac{T_{p,i}}{{\vartheta }_p}|>\epsilon ,\mathbf{\Theta })=\frac{1}{{\vartheta }_p^2}\sum _{i=1}^{p}E$$

$$({\zeta }_{n,i}^2{(w_{n,i}-1)}^2;|w_{n,i}-1|>\epsilon {\vartheta }_p/{\zeta }_{n,i}\le \frac{1}{{\vartheta }_p^2}\sum _{i=1}^{p}E$$

$$\begin{gathered} ({\zeta }_{n,1}^2{(w_{n,i}-1)}^2;|w_{n,i}-1|>\epsilon {\vartheta }_p/{\zeta }_{n,p},\mathbf{\Theta }) \\ \le \frac{p}{{\vartheta }_p^2}E \end{gathered}$$

$$({\zeta }_{n,1}^2{(w_{n,i}-1)}^2;|w_{n,i}-1|>\epsilon \sqrt{2p}{\zeta }_{n,1}/{\zeta }_{n,p},\mathbf{\Theta })\le \frac{1}{2{\zeta }_{n,n}^2}E$$

$$({\zeta }_{n,1}^2{(w_{n,i}-1)}^2;|w_{n,i}-1|>\epsilon \sqrt{2p}{\zeta }_{n,1}/{\zeta }_{n,p},\mathbf{\Theta })\to 0,$$

Here E(x; a, b) denotes the expected value of x with respect to condition a and b. The last inequality holds as {ζ_n,i}_i are bounded away from both zero and infinity asymptotically (Lemma 3). Then, according to Lindeberg-Feller theorem (Lindeberg 1922; Feller 1971), we have $\frac{{\sum }_{i=1}^p{T}_{p,i}}{{\vartheta }_p}|\mathbf{\Theta }\Rightarrow N(0,1)$. It follows $\frac{\frac{1}{p}RHT(\lambda )-\frac{1}{p}{\sum }_{i=1}^p{\zeta }_{n,i}}{{(\frac{2}{p^2}{\sum }_{i=1}^p{\zeta }_{n,i}^2)}^{1/2}}\Rightarrow N(0,1)$, based on the following:

Lemma 4 ${\{x_n,y_n\}}_{n=1}^{\infty }$ are random variables. g_n(x_n, y_n) is a real function of x_n and y_n. If g_n|y_n → _D Z, where Z is independent of {y_n}, then we have g_n → _D Z.

The proof of Lemma 4 is straightforward and is provided in the Supplementary Information.

5.5 Proof of Lemma 2

Proof of Lemma 2 is based on the ideas developed in (Ledoit and Péché 2010).

Proof of (19): Denote the resolvent of S_n by R_n(z) = (S_n−zI_p)⁻¹. Note that, ${\mathbf{S}}_n={\sum }_{j=1}^n{\mathbf{\Sigma }}_p^{1/2}{\mathbf{Y}}_j{\mathbf{Y}}_j^T{\mathbf{\Sigma }}_p^{1/2}$ where $\sqrt{n}{\mathbf{Y}}_j~N(\mathbf{0},{\mathbf{I}}_p)$. Hence, for any j ∈ {1, …, n},

$${\mathbf{R}}_n(z)={(({\mathbf{S}}_n-{\mathbf{\Sigma }}_p^{1/2}{\mathbf{Y}}_j{\mathbf{Y}}_j^T{\mathbf{\Sigma }}_p^{1/2})-z{\mathbf{I}}_p+{\mathbf{\Sigma }}_p^{1/2}{\mathbf{Y}}_j{\mathbf{Y}}_j^T{\mathbf{\Sigma }}_p^{1/2})}^{-1}={\mathbf{R}}_n^{(j)}(z)-\frac{{\mathbf{R}}_n^{(j)}(z){\mathbf{\Sigma }}_p^{1/2}{\mathbf{Y}}_j{\mathbf{Y}}_j^T{\mathbf{\Sigma }}_p^{1/2}{\mathbf{R}}_n^{(j)}(z)}{1+{\mathbf{Y}}_j^T{\mathbf{\Sigma }}_p^{1/2}{\mathbf{R}}_n^{(j)}(z){\mathbf{\Sigma }}_p^{1/2}{\mathbf{Y}}_j}$$ (27)

where ${\mathbf{R}}_n^{(j)}(z)={({\mathbf{S}}_n-{\mathbf{\Sigma }}_p^{1/2}{\mathbf{Y}}_j{\mathbf{Y}}_j^T{\mathbf{\Sigma }}_p^{1/2}-z{\mathbf{I}}_p)}^{-1}$. Observe that Y_j and ${\mathbf{R}}_n^{(j)}(z)$ are independent for all z ∈ ℂ.

Post-multiplying both sides of the identity (S_n − zI_p) + zI_p = S_n by R_n(z), we have

$${\mathbf{I}}_p+z{\mathbf{R}}_n(z)={\mathbf{S}}_n{\mathbf{R}}_n(z)=\sum _{j=1}^n{\mathbf{\Sigma }}_p^{1/2}{\mathbf{Y}}_j{\mathbf{Y}}_j^T{\mathbf{\Sigma }}_p^{1/2}{\mathbf{R}}_n(z).$$ (28)

Now, letting z = −λ (for which R_n(z) is well defined). Taking trace on both sides of (28), dividing both sides by p, and using (27), we have

$$1-\lambda \frac{1}{p}\text{tr}({\mathbf{R}}_n(-\lambda ))=\frac{1}{p}\sum _{j=1}^n[{\mathbf{Y}}_j^T{\mathbf{\Sigma }}_p^{1/2}{\mathbf{R}}_n^{(j)}(-\lambda ){\mathbf{\Sigma }}_p^{1/2}{\mathbf{Y}}_j-\frac{{({\mathbf{Y}}_j^T{\mathbf{\Sigma }}_p^{1/2}{\mathbf{R}}_n^{(j)}(-\lambda ){\mathbf{\Sigma }}_p^{1/2}{\mathbf{Y}}_j)}^2}{1+{\mathbf{Y}}_j^T{\mathbf{\Sigma }}_p^{1/2}{\mathbf{R}}_n^{(j)}(-\lambda ){\mathbf{\Sigma }}_p^{1/2}{\mathbf{Y}}_j}]=\frac{1}{p}\sum _{j=1}^n\frac{{\mathbf{Y}}_j^T{\mathbf{\Sigma }}_p^{1/2}{\mathbf{R}}_n^{(j)}(-\lambda ){\mathbf{\Sigma }}_p^{1/2}{\mathbf{Y}}_j}{1+{\mathbf{Y}}_j^T{\mathbf{\Sigma }}_p^{1/2}{\mathbf{R}}_n^{(j)}(-\lambda ){\mathbf{\Sigma }}_p^{1/2}{\mathbf{Y}}_j}=\frac{n}{p}-\frac{1}{p}\sum _{j=1}^n\frac{1}{1+{\mathbf{Y}}_j^T{\mathbf{\Sigma }}_p^{1/2}{\mathbf{R}}_n^{(j)}(-\lambda ){\mathbf{\Sigma }}_p^{1/2}{\mathbf{Y}}_j}=\frac{n}{p}-\frac{1}{p}\sum _{j=1}^n\frac{1}{1+\frac{1}{n}\text{tr}({\mathbf{\Sigma }}_p^{1/2}{\mathbf{R}}_n^{(j)}(-\lambda ){\mathbf{\Sigma }}_p^{1/2})}+{\delta }_n^{(1)}=\frac{n}{p}-\frac{n}{p}\frac{1}{1+\frac{p}{n}\frac{1}{p}\text{tr}({\mathbf{R}}_n(-\lambda ){\mathbf{\Sigma }}_p)}+{\delta }_n^{(1)}+{\delta }_n^{(2)}$$ (29)

for appropriate residual terms ${\delta }_n^{(1)}$ and ${\delta }_n^{(2)}$. Also, define

$${\delta }_n^{(3)}=\frac{1}{p}\text{tr}({\mathbf{R}}_n(-\lambda ))-m_F(-\lambda ).$$ (30)

We claim that (proved in Supplementary Information)

$$\underset{1\le k\le 3}{\text{max}}\sqrt{n}|{\delta }_n^{(k)}|\stackrel{P}{\to }0,\mathrm{a}\mathrm{s}n\to \infty .$$ (31)

Define ${\delta }_n=(p/n)({\delta }_n^{(1)}+{\delta }_n^{(2)}+\lambda {\delta }_n^{(3)})$. Then, from (29) and (30) we can express

$$\frac{1}{p}\text{tr}({\mathbf{R}}_n(-\lambda ){\mathbf{\Sigma }}_p)=\frac{n}{p}(\frac{1}{1-\frac{p}{n}(1-\lambda m_F(-\lambda ))+{\delta }_n}-1)=\frac{{\mathbf{\Theta }}_1(\lambda ,\gamma )-\frac{(n/p){\delta }_n}{1-\gamma (1-\lambda m_F(-\lambda ))}}{1+\frac{{\delta }_n}{1-\gamma (1-\lambda m_F(-\lambda ))}+(\gamma -\frac{p}{n}){\mathbf{\Theta }}_1(\lambda ,\gamma )}={\mathbf{\Theta }}_1(\lambda ,\gamma )+o_P(n^{-1/2}).$$

The second equality is using (7) and the last equality is by (31) and assumption A2.

Proof of (20): Post-multiplying the identity (28) (with z = −λ) by Σ_pR_n(−λ), taking trace, dividing by p, and using (27), we have

$$\frac{1}{p}\text{tr}({\mathbf{R}}_n(-\lambda ){\mathbf{\Sigma }}_p)-\lambda \frac{1}{p}\text{tr}({({\mathbf{R}}_n(-\lambda ))}^2{\mathbf{\Sigma }}_p)=\frac{1}{p}\sum _{j=1}^n{\mathbf{Y}}_j^T{\mathbf{\Sigma }}_p^{1/2}{\mathbf{R}}_n(-\lambda ){\mathbf{\Sigma }}_p{\mathbf{R}}_n(-\lambda ){\mathbf{\Sigma }}_p^{1/2}{\mathbf{Y}}_j=\frac{1}{p}\sum _{j=1}^n[{\mathbf{Y}}_j^T{\mathbf{\Sigma }}_p^{1/2}{\mathbf{R}}_n^{(j)}(-\lambda ){\mathbf{\Sigma }}_p{\mathbf{R}}_n(-\lambda ){\mathbf{\Sigma }}_p^{1/2}{\mathbf{Y}}_j-\frac{({\mathbf{Y}}_j^T{\mathbf{\Sigma }}_p^{1/2}{\mathbf{R}}_n^{(j)}(-\lambda ){\mathbf{\Sigma }}_p^{1/2}{\mathbf{Y}}_j)({\mathbf{Y}}_j^T{\mathbf{\Sigma }}_p^{1/2}{\mathbf{R}}_n^{(j)}(-\lambda ){\mathbf{\Sigma }}_p{\mathbf{R}}_n(-\lambda ){\mathbf{\Sigma }}_p^{1/2}{\mathbf{Y}}_j)}{1+{\mathbf{Y}}_j^T{\mathbf{\Sigma }}_p^{1/2}{\mathbf{R}}_n^{(j)}(-\lambda ){\mathbf{\Sigma }}_p^{1/2}{\mathbf{Y}}_j}]=\frac{1}{p}\sum _{j=1}^n\frac{{\mathbf{Y}}_j^T{\mathbf{\Sigma }}_p^{1/2}{\mathbf{R}}_n^{(j)}(-\lambda ){\mathbf{\Sigma }}_p{\mathbf{R}}_n(-\lambda ){\mathbf{\Sigma }}_p^{1/2}{\mathbf{Y}}_j}{1+{\mathbf{Y}}_j^T{\mathbf{\Sigma }}_p^{1/2}{\mathbf{R}}_n^{(j)}(-\lambda ){\mathbf{\Sigma }}_p^{1/2}{\mathbf{Y}}_j}=\frac{1}{p}\sum _{j=1}^n\frac{{\mathbf{Y}}_j^T{\mathbf{\Sigma }}_p^{1/2}{\mathbf{R}}_n^{(j)}(-\lambda ){\mathbf{\Sigma }}_p{\mathbf{R}}_n^{(j)}(-\lambda ){\mathbf{\Sigma }}_p^{1/2}{\mathbf{Y}}_j}{{(1+{\mathbf{Y}}_j^T{\mathbf{\Sigma }}_p^{1/2}{\mathbf{R}}_n^{(j)}(-\lambda ){\mathbf{\Sigma }}_p^{1/2}{\mathbf{Y}}_j)}^2}=\frac{1}{p}\sum _{j=1}^n\frac{\frac{1}{n}\text{tr}({\mathbf{\Sigma }}_p^{1/2}{\mathbf{R}}_n^{(j)}(-\lambda ){\mathbf{\Sigma }}_p{\mathbf{R}}_n^{(j)}(-\lambda ){\mathbf{\Sigma }}_p^{1/2})}{{(1+\frac{1}{n}\text{tr}({\mathbf{\Sigma }}_p^{1/2}{\mathbf{R}}_n^{(j)}(-\lambda ){\mathbf{\Sigma }}_p^{1/2}))}^2}+{\delta }_n^{(4)}=\frac{\frac{1}{p}\text{tr}({\mathbf{R}}_n(-\lambda ){\mathbf{\Sigma }}_p{\mathbf{R}}_n(-\lambda ){\mathbf{\Sigma }}_p)}{{(1+\frac{p}{N}\frac{1}{p}\text{tr}({\mathbf{R}}_n(-\lambda ){\mathbf{\Sigma }}_p))}^2}+{\delta }_n^{(4)}+{\delta }_n^{(5)}$$ (32)

for appropriate residuals ${\delta }_n^{(4)}$ and ${\delta }_n^{(5)}$. Define

$${\delta }_n^{(6)}=-\frac{1}{p}\text{tr}({({\mathbf{R}}_n(-\lambda ))}^2{\mathbf{\Sigma }}_p)-\frac{\partial }{\partial \lambda }{\mathbf{\Theta }}_1(\lambda ,\gamma )=\frac{d}{d\lambda }[\frac{1}{p}\text{tr}({\mathbf{R}}_n(-\lambda ){\mathbf{\Sigma }}_p)]-\frac{\partial }{\partial \lambda }{\mathbf{\Theta }}_1(\lambda ,\gamma ).$$ (33)

We claim that (proved in Supplementary Information)

$$\underset{4\le k\le 6}{\text{max}}|{\delta }_n^{(k)}|\stackrel{P}{\to }0,\mathrm{a}\mathrm{s}n\to \infty .$$ (34)

It then follows from (29), (31), (32), (34) and (8) that

$$\frac{1}{p}\text{tr}({\mathbf{R}}_n(-\lambda ){\mathbf{\Sigma }}_p{\mathbf{R}}_n(-\lambda ){\mathbf{\Sigma }}_p)={\mathbf{\Theta }}_2(\lambda ,\gamma )+o_P(1).$$

6 SUMMARY

Pathway analysis is a useful tool for genomics or proteomics array studies. By combining information of multiple genes or proteins from the same functional group, pathway analysis may achieve better power for detecting meaningful biological signals and the result may augment the interpretation provided by univariate analyses.

Many methods have been proposed for pathway (or gene-set) analysis in the literature, mostly for microarray gene expression studies and for genome-wide association studies. Proteomics, as an emerging technology, has distinctive features compared to other ‘omics’ studies. Proteomics data may have complicated correlation structure, substantial missingness and small sample size. Furthermore, depending on the proteomic platform, the proteomic data may consist of concentration ratios rather than abundant concentrations. These factors limit the suitability of existing statistical methods.

When considering multiple proteins together, some multivariate tests can be used to handle correlation. However, when the sample size is small, and is close to or less than the number of proteins considered, these tests experience difficulty. Some novel tests have been proposed for the p ≥ n scenarios, but they rely on asymptotic theory and evidently require a relatively large sample size for good performance. In addition, missing data further complicates the use of these tests. In this work, we propose a new RHT test, in which we design the test statistics using regularization techniques, and propose a non-parametric testing procedure to conservatively control the type I error rate. The regularization parameter is adaptively selected based on the data, thereby enhancing the robustness of the proposed method.

Another contribution of the paper is to establish the asymptotic theory for the proposed RHT statistics under a diverging dimension regime, such that p/n → γ ∈ (0,∞). These results provide useful insights on the properties of the test statistics, and suggest that RHT has intrinsic advantages over the original HT in terms of power when p is close to or greater than n. In the extensive simulation studies, we further demonstrate that RHT has favorable performance compared to existing methods, especially when missing data and/or correlations are present.

Our nonparametric testing procedure based on resampling makes a missing completely at random assumption. MCAR is a reasonable working assumption for this application. However, the pattern of missingness can be due to many complicated factors in mass spectrometry experiments, and MCAR may not apply for mass spectrometry based data more generally. Further developments to relax the missing data assumption will be useful and additional research is underway. On the other hand, even under the MCAR assumption, the missing data raises challenges for statistical analysis. Particularly, for test statistics in complicated forms (e.g. T_D, T_BS and F⁺ tests), their distributions vary substantially according to the degrees and the patterns of missingness, and thus tend to be less robust in application. On the other hand, the simple quadratic form of the proposed RHT makes its performance less affected by various missing data patterns. In addition, we develop a resampling-based testing procedure for RHT, in which the pattern of missingness is explicitly accounted for, and the cut-off values are obtained empirically, as opposed to relying entirely on the asymptotic theory for the complete data scenario.

We apply the RHT test to a hormone therapy proteomic study and identify five pathways whose protein serum concentrations changed after one year of hormone therapy treatment. These findings may provide additional insight for understanding the biological processes mediating the clinical effects of hormone therapy treatment (Pitteri et al. 2009).

The R package RHT is available from the authors upon request. It will be made available through CRAN shortly.