Testing for inequality constraints in singular models by trimming or winsorizing the variance matrix

Abstract

There are many applications in which a statistic follows, at least asymptotically, a normal distribution with a singular or nearly singular variance matrix. A classic example occurs in linear regression models under multicollinearity but there are many more such examples. There is well–developed theory for testing linear equality constraints when the alternative is two–sided and the variance matrix is either singular or non–singular. In recent years there is considerable, and growing, interest in developing methods for situations in which the estimated variance matrix is nearly singular. However, there is no corresponding methodology for addressing one–sided, i.e., constrained or ordered alternatives. In this paper we develop a unified framework for analyzing such problems. Our approach may be viewed as the trimming or winsorizing of the eigenvalues of the corresponding variance matrix. The proposed methodology is applicable to a wide range of scientific problems and to a variety of statistical models in which inequality constraints arise. We illustrate the methodology using data from a gene expression microarray experiment obtained from the NIEHS’ Fibroid Growth Study.

1. Introduction

A common problem in many applications is to compare two or more ordered experimental groups in terms of one or more outcome variables. For example, a toxicologist may be interested in comparing different dose groups in terms of several outcomes such as body weight, red blood cell count, hematocrits and so forth. Often the statistical problem reduces to drawing inferences regarding a parameter $\mathit{\theta }\in {ℝ}^p$ using a statistic S_n which, under suitable standardization and regularity conditions, is asymptotically normally distributed, i.e.,

$$\sqrt{n}\left({\mathit{S}}_n-\mathit{\theta }\right)\Rightarrow {\mathcal{N}}_p(\mathbf{0},\mathbf{\Sigma })$$ (1.1)

as n → ∞ where ⇒ denotes convergence in distribution. The statistic S_n may be a vector of differences among means, a collection of rank statistics or an estimator of a regression parameter. In general, Σ is unknown and must be estimated from the data; furthermore Σ may be non–singular, singular or nearly singular. By nearly singular we mean that its condition number, i.e., the ratio of its largest to its smallest eigenvalue, is extremely large. As an example, near singularity arises in regression models due to multicollinearity (Silvey 1969, Montgomery and Peck, 2007). It is well known that common statistical methods, especially those based on matrix inversion, perform poorly when Σ is nearly singular and can not be directly used when Σ is singular.

Unconstrained statistical inference when the underlying variance matrix is singular is well studied in the context of linear regression (Khatri 1968, Rao and Mitra 1971 and Rao 1972), where Σ is typically known up to a constant. However, in many applications Σ must be estimated from the data. Moreover, Σ and/or its estimator denoted here by Σ_n, may be singular and, additionally, their rank may not be known in advance. Unconstrained Inference in such settings has been addressed by numerous authors in the statistical and econometric literature. Some examples include Moore (1977), Andrews (1987), Hadi and Wells (1990, 1991), Lutkepohl and Burda (1997), Dufour and Valery (2015) and Duplinskiy (2015). We note that in nonlinear regression singularity or near singularity of the variance matrix is rather common. For example, in the context of quantitative high throughput screening (qHTS) assays, where thousands of chemicals are evaluated for toxicity using cell lines, Lim et al. (2012) noticed that the condition number of the information matrix may be very large; sometimes as large as 10⁹ or higher. Hadi and Wells (1990) provide examples of nonlinear models for which the information matrix is singular for some values of the model parameters; in fact, these are often the values specified by the null hypothesis. Another mechanism by which singular variance matrices arise is when θ = h(η), η is estimated by ${\widehat{\mathit{\eta }}}_n$ and the Jacobian of h is rank deficient. This leads, by the δ–method, to a singular variance matrix.

When the experimental groups are naturally ordered researchers are often interested in performing one–sided tests in which case the parameter space under the alternative hypothesis is not a linear subspace but a convex cone, denoted by $\mathcal{C}$. Tests for equality against an ordering are often formalized as

$$H_0:\mathit{\theta }\in \mathcal{L}\text{versus}{H}_1:\mathit{\theta }\in \mathcal{C}\backslash \mathcal{L},$$ (1.2)

where $\mathcal{L}$ is a linear subspace and $\mathcal{L}\subset \mathcal{C}\subset {ℝ}^p$. In practice $\mathcal{L}$ is often the singleton {0} and $\mathcal{C}$ is often defined by a finite set of linear inequalities. The theory for such testing problems is well developed, cf. Silvapulle and Sen (2005), when both Σ and its estimator Σ_n are non–singular. Models, both parametric and nonparametric, in which the variance matrix is known to be singular and in which testing (1.2) is of interest arise in a wide range of scientific problems, such as dose response studies. A concrete example will be discussed in more detail later on. We are not aware of a general theory or methodology for ordered inference when Σ and/or Σ_n are singular or nearly singular. In this paper we address this open class of problems in a principled manner. Pioneered by Dixon (1960) and Tukey (1962), trimming and winsorizing data are two well–known strategies for dealing with extreme observations, or outliers, and are widely used in robust statistics (Huber and Ronchetti, 2009). Inspired by the use of trimming and winsorizing in classical data analysis, in this paper we develop trimming and winsorizing based approaches to address near singularities in multivariate data. In this setting the role of outliers is played by eigenvalues that are associated with a large condition number, i.e., the eigenvalues which are different from zero but much smaller than the largest eigenvalue. These are exactly the eigenvalues which create challenges when analyzing multivariate data. Our proposed methodology builds on and extends the existing methods developed for unconstrained singular models as referenced earlier.

The paper is organized in the following way. In Section 2 we develop a modified likelihood ratio test assuming a known singular variance matrix. In Section 3 we extend the methodology to the case of unknown, possibly singular, or nearly singular variance matrices by introducing trimmed and winsorized tests. Since these two tests depend upon a threshold parameter, specified by the user, we modify them by introducing corresponding data–driven supremum type tests which do not require any input from the data analyst. The performance of the proposed tests in terms of their type I errors and powers was evaluated using an extensive simulation study. The study design and results are summarized in Section 4. In Section 5 we describe the NIEHS Fibroid Growth Study (FGS) of Peddada et al. (2008) and illustrate our proposed methodology by reanalyzing gene expression data obtained in FGS. Although there have been studies in which single genes have been correlated with tumor size (cf. Grandhi et al., 2016), to the best of our knowledge there are no studies in which a collection of genes, or a pathway, were correlated with tumor size; thus, as far as we know, this is the first paper that performs such a multivariate analysis. Section 6 provides a brief summary and further discussion. Additional results, as well as all proofs, are provided in the online supplementary text.

2. Known singular variance matrix

Since Σ is symmetric and nonnegative definite we may write its spectral decomposition as Σ = EΛE^T where E is an orthogonal matrix whose columns are the eigenvectors of Σ and is a diagonal matrix with nonnegative elements λ₁, …, λ_p which are the eigenvalues of Σ. Without loss of generality assume that λ₁ ≥ λ₂ ≥ … ≥ λ_p. Moreover if Σ is singular then q = rank(Σ) < p and thus λ_i > 0 when i ≤ p − q and λ_i = 0 when i ≥ p − q + 1. It is well known that the Moore–Penrose generalized inverse of Σ denoted by Σ⁺ is equal to E^TΛ⁺E where Λ⁺ is a diagonal matrix with elements 1/λ_i if i ≤ p – q and 0 otherwise. Let M be the q × p matrix obtained by dropping the last p − q rows of the matrix (Λ⁺)^1/2E^T which are easily verified to be identically zero. Finally let $\mathcal{S}(\mathbf{\Sigma })$ denote the column space of Σ. We shall index by ‘*’ the intersection of any set with $\mathcal{S}(\mathbf{\Sigma })$. Thus we define ${\mathcal{L}}_{*}=\mathcal{L}\cap \mathcal{S}(\mathbf{\Sigma })$, ${\mathcal{C}}_{*}=\mathcal{C}\cap \mathcal{S}(\mathbf{\Sigma })$ and ${\mathcal{R}}_{*}=\mathcal{R}\cap \mathcal{S}(\mathbf{\Sigma })$ where $\mathcal{R}=\mathcal{C}\cap {\mathcal{L}}^{\perp }$ and ${\mathcal{L}}^{\perp }$ is the orthogonal complement of $\mathcal{L}$. Clearly ${\mathcal{L}}_{*}$ is a linear space, and ${\mathcal{C}}_{*}$ and ${\mathcal{R}}_{*}$ are convex cones.

The following theorem defines, and then describes, the distribution of the modified likelihood ratio test (mLRT) assuming a normal model with a known singular variance matrix. We use the qualification modified for the LRT because the singular normal distribution does not have a density and therefore the LRT does not exist.

Theorem 2.1

Consider testing (1.2) when $\mathit{Y}\sim {\mathcal{N}}_p(\mathit{\theta },\mathbf{\Sigma })$ and Σ is known and singular. Assume that 0 ∈ support(Y). Then, if ${\mathcal{C}}_{*}\ne \left\{\mathbf{0}\right\}$ the mLRT exists and is given by

$$T={\mathit{Y}}^T{\mathbf{\Sigma }}^{+}\mathit{Y}-\min \left\{{(\mathit{Y}-\mathit{\theta })}^T{\mathbf{\Sigma }}^{+}(\mathit{Y}-\mathit{\theta }):\mathit{\theta }\in {\mathcal{R}}_{*}\right\}$$ (2.1)

Furthermore in the notation of Silvapulle and Sen (2005), under the null the statistic (2.1) is distributed as ${\overline{\chi }}^2\left({\mathit{I}}_q,\mathit{M}{\mathcal{R}}_{*}\right)$ which is defined by

$${\mathit{X}}^T\mathit{X}-\min \left\{{(\mathit{X}-\mathit{\eta })}^T(\mathit{X}-\mathit{\eta }):\mathit{\eta }\in \mathit{M}{\mathcal{R}}_{*}\right\},$$ (2.2)

where X is a ${\mathcal{N}}_q\left(\mathit{\eta },{\mathit{I}}_q\right)$ random variable (RV) and I_q is the q × q identity matrix.

Theorem 2.1 generalizes several results published in the literature. Firstly, it generalizes Theorem 3.7.1 in Silvapulle and Sen (2005) to the case where Σ is not a full rank matrix. Next it generalizes Lemmas 1 and 2 in Moore (1977), originally discovered by Khatri (1968) and Rao and Mitra (1971), from two–sided to one–sided hypotheses. In the proof of Theorem 2.1, given in the supplementary materials, it is shown that there are two equivalent forms for the mLRT, (2.1) and (2.2); the distribution of the mLRT is an immediate consequence of this equivalence. In other words it is shown that testing (1.2) with a singular Σ is equivalent to testing H₀ : η = 0 versus $H_1:\mathit{\eta }\in \mathit{M}{\mathcal{R}}_{*}$ based on a random variable (RV) X distributed as ${\mathcal{N}}_q(\mathit{\eta },\mathit{I})$ where $\mathit{M}{\mathcal{R}}_{*}=\left\{\mathit{M}\mathit{\theta }:\mathit{\theta }\in {\mathcal{R}}_{*}\right\}$ is a cone. Thus, the original problem is solved by the usual procedure but in a lower dimensional space. Effectively the test is carried out as if $\mathcal{L}\subset \mathcal{C}\subset \mathcal{S}(\mathbf{\Sigma })$ but we need not specify that in advance. This feature is of practical importance when Σ is unknown as in Section 3. It is also important to note that Theorem 2.1 remains valid if we replace the Moore–Penrose inverse Σ⁺ with any other generalized inverse Σ⁻ and choose instead of M, as defined above, any q × p matrix N which satisfies Σ⁻ = N^T N.

The statistic (2.1) may be computed using low–rank quadratic programming (cf. Boyd and Vandenberghe, 2004) where the cone ${\mathcal{R}}_{*}$ is given in Proposition 2.1 below. In some applications there may be an interest in computing (2.2) directly, which requires expressing the cone $\mathit{M}{\mathcal{R}}_{*}$ in standard form, that is as $\left\{\mathit{\eta }\in {ℝ}^q:\mathit{D}\mathit{\eta }\ge \mathbf{0}\right\}$ for some matrix D. In some cases the cone $\mathit{M}{\mathcal{R}}_{*}$ may be found by simple and direct analysis. In general, though, this is not the case. Fortunately, a broadly applicable scheme for explicitly finding the inequalities defining $\mathit{M}{\mathcal{R}}_{*}$ is available and known as the double description method (Avis and Fukuda, 1992). Explaining the underlying key ideas requires some additional notation. Assume for now that $\mathcal{C}$ is a polyhedral pointed convex cone, i.e., $\mathcal{C}=\left\{\mathit{\theta }\in {ℝ}^p:\mathit{C}\mathit{\theta }\ge \mathbf{0}\right\}$ for some restriction matrix C with a finite number of rows. This form is referred to as the H–representation of $\mathcal{C}$ (H for half–planes). Alternatively, by the Minkowski–Weyl Theorem (cf. Proposition 3.12.10 in Silvapulle and Sen 2005), $\mathcal{C}$ can be also expressed as

$$\mathcal{C}=\mathrm{cone}\left({\mathit{\theta }}_1,\dots ,{\mathit{\theta }}_m\right)=\left\{{\lambda }_1{\mathit{\theta }}_1+\cdots +{\lambda }_m{\mathit{\theta }}_m:{\lambda }_1,\dots ,{\lambda }_m\ge 0\right\}$$

where the set {θ₁, …, θ_m} contains the extreme rays of $\mathcal{C}$. This form is referred to as the V–representation of $\mathcal{C}$ (V for vertices). Recall that an extreme ray in a convex cone $\mathcal{C}$ is an element $\mathit{\theta }\in \mathcal{C}$, such that if θ = θ₁ + θ₂ with ${\mathit{\theta }}_1,{\mathit{\theta }}_2\in \mathcal{C}$ then θ = λθ₁ or θ = λθ₂ for some λ ≥ 0. The set of extreme rays of $\mathcal{C}$ is denoted by $\mathrm{extr}(\mathcal{C})$. Clearly if θ is an extreme ray so is λθ for all λ ≥ 0, thus for convenience we will pick only one element from this equivalence class. Since $\mathcal{C}=\text{cone}(\mathrm{extr}(\mathcal{C}))$ the extreme rays are those elements which are necessary for generating the cone. Both the H–representation and the V–representation may be redundant, i.e., the matrix C may include unnecessary rows and the set {θ₁, …, θ_m} may contain unnecessary rays. Such redundancies can always be removed. The conversion from the H–representation to the V–representation is know as the vertex enumeration problem and the reverse as the facet enumeration problem, see Lauritzen (2011) for a beautiful account of the theory which involves polyhedral geometry and computational linear algebra. Both conversions are used for computing the standard form of the cone $\mathit{M}{\mathcal{R}}_{*}$.

Proposition 2.1

Let $\mathcal{C}=\left\{\mathit{\theta }\in {ℝ}^p:\mathit{C}\mathit{\theta }\ge \mathbf{0}\right\}$ and let L and S be matrices, the rows of which, are bases for $\mathcal{L}$ and $\mathcal{S}{(\mathbf{\Sigma })}^{\perp }$ respectively. Then

$$\mathit{M}{\mathcal{R}}_{*}=\mathrm{cone}\left(\mathrm{extr}(\mathit{M}\mathit{\theta }):\mathit{\theta }\in \mathrm{extr}\left({\mathcal{R}}_{*}\right)\right)$$

where ${\mathcal{R}}_{*}=\left\{\mathit{\theta }\in {ℝ}^p:{\mathit{R}}_{*}\mathit{\theta }\ge \mathbf{0}\right\}$ and ${\mathit{R}}_{*}^T=\left({\mathit{C}}^T,{\mathit{L}}^T,-{\mathit{L}}^T,{\mathit{S}}^T,-{\mathit{S}}^T\right)$.

By Proposition 2.1 $\mathit{M}{\mathcal{R}}_{*}$ is the conic combination of the finite set extr(Mθ) where $\mathit{\theta }\in \mathrm{extr}\left({\mathcal{R}}_{*}\right)$. The H–representation for ${\mathcal{R}}_{*}$ is given in Proposition 2.1, hence finding $\mathrm{extr}\left({\mathcal{R}}_{*}\right)$ requires the solution of a vertex enumeration problem. Proposition 2.1 provides a V–representation of $\mathit{M}{\mathcal{R}}_{*}$ from which an H–representation is found via a facet enumeration problem. In some settings intermediate, redundancy removal steps, may be applied. Although Proposition 2.1 discusses only convex polyhedral cones, with suitable modifications, it can be generalized to arbitrary convex cones.

Finally, it is well known that if $T\sim {\overline{\chi }}^2\left({\mathit{I}}_q,\mathit{M}{\mathcal{R}}_{*}\right)$ then

$$\mathbb{P}(T\ge c)=\sum _{j=1}^q{w}_j\mathbb{P}({\chi }_j^2\ge c)$$

where $w_j=w_j\left({\mathit{I}}_q,\mathit{M}{\mathcal{R}}_{*}\right)$, j = 0, …, q are nonnegative weights that sum to unity and are functions of the constraints specified in the alternative hypothesis. The weights can be computed in closed form, when q ≤ 4 and $\mathit{M}{\mathcal{R}}_{*}$ is the orthant, or some other simple cone. Otherwise the weights or the entire distribution may be estimated by simulation as discussed in detail by Silvapulle and Sen (2005).

3. Unknown and possibly singular variance matrix

Theorem 2.1 shows how to test (1.2) when (1.1) holds and Σ is a fixed known singular matrix. However, in most situations Σ is unknown and thus needs to be estimated along with θ Let Σ_n denote an estimator of Σ which is assumed to be symmetric and nonnegative definite. We will further assume that Σ_n is consistent and that

$$b_n\left({\mathbf{\Sigma }}_n-\mathbf{\Sigma }\right)\Rightarrow \mathit{Q}$$ (3.1)

where Q is a matrix valued RV and b_n → ∞ is a sequence of constants; usually $b_n=\sqrt{n}$. Note that Σ_n need not be singular for some, or even all finite n, even if Σ is singular. Conversely Σ_n may be singular even if Σ is not. As noted in Andrews (1987), if rank(Σ_n) ≠ rank(Σ) then quadratic forms associated with Σ_n may not converge to those associated with Σ due to the discontinuity of generalized inverses, i.e., ${\mathit{X}}_n^T{\mathbf{\Sigma }}_n^{+}{\mathit{X}}_n$ may not converge to X^TΣ⁺X p even if X_n ⇒ X and ${\mathbf{\Sigma }}_n\stackrel{p}{\to }\mathbf{\Sigma }$. Hence, quadratic form based statistics, such as projected tests common in order restricted inference, may not converge to their assumed limits.

3.1. Trimmed tests

To overcome the above noted problem we trim the variance matrix by dropping all eigenvalues smaller than some pre–specified ε. Formally,

Definition 3.1

Let ε > 0. We refer to the matrix Σ_T(ε) = EΛ_T(ε)E^T, where Λ_T(ε), which is the diagonal matrix obtained from Λ by replacing any element of Λ which is smaller than ε by 0, as the ε–trimmed variance matrix. Its Moore–Penrose inverse, i.e., ${\mathit{E}}^T{\mathbf{\Lambda }}_T^{+}(\epsilon )\mathit{E}$ is referred to as the ε–trimmed inverse of Σ and denoted by Σ⁺(ε).

For convenience we now drop the subscript T whenever possible. Let Σ⁺(ε) denote the trimmed variance, then its Moore–Penrose inverse, i.e., the matrix ${\mathbf{\Sigma }}_n^{+}(\epsilon )$ equals ${\mathit{E}}_n^T{\mathbf{\Lambda }}_n^{+}(\epsilon ){\mathit{E}}_n$ where ${\mathbf{\Lambda }}_n^{+}(\epsilon )$ is a diagonal matrix whose elements are 1= λ_i,n if λ_i,n ≥ ε and 0 otherwise. Dufour and Valery (2015) refer to ${\mathbf{\Sigma }}_n^{+}(\epsilon )$ as the spectral cut–off Moore–Penrose inverse. We now introduce a distance based test for (1.2) which is motivated by the mLRT for normal means given in (2.1) and the above definition, i.e., for each n we de ne the trimmed constrained test statistic as

$$T_n\left({\epsilon }_n\right)=\min \left\{n{\left({\mathit{S}}_n-\mathit{\theta }\right)}^T{\mathbf{\Sigma }}_n^{+}\left({\epsilon }_n\right)\left({\mathit{S}}_n-\mathit{\theta }\right):\mathit{\theta }\in {\mathcal{L}}_{*}^n\right\}-\min \left\{n{\left({\mathit{S}}_n-\mathit{\theta }\right)}^T{\mathbf{\Sigma }}_n^{+}\left({\epsilon }_n\right)\left({\mathit{S}}_n-\mathit{\theta }\right):\mathit{\theta }\in {\mathcal{C}}_{*}^n\right\}$$ (3.2)

where ${\mathcal{L}}_{*}^n={\mathcal{L}}_{*}\left({\epsilon }_n\right)=\mathcal{L}\cap \mathcal{S}\left({\mathbf{\Sigma }}_n\left({\epsilon }_n\right)\right)$, ${\mathcal{C}}_{*}^n={\mathcal{C}}_{*}\left({\epsilon }_n\right)=\mathcal{C}\cap \mathcal{S}\left({\mathbf{\Sigma }}_n\left({\epsilon }_n\right)\right)$ and $\mathcal{S}\left({\mathbf{\Sigma }}_n\left({\epsilon }_n\right)\right)$ is the column space of Σ_n (ε_n). Note that by definition T_n(ε_n) is a function of the threshold ε_n. Since in general the value of the smallest eigenvalue may not be known in advance it is appropriate to allow ε_n to vary with n. If ε_n → 0 we denote T_n(ε_n) by T_n. The limiting distribution of T_n is given below; a discussion of the statistic T_n(ε) for a fixed ε is deferred to Section 3.2.

Theorem 3.1

Consider testing (1.2). Suppose that (1.1) holds and the limit distribution satisfies the conditions of Theorem 2.1. In additions suppose that Σ_n satisfies (3.1). Then under H₀ and for any sequence of cut–off values {Σ_n} satisfying ε_n → 0 and b_nε_n → ∞ we have

$$T_n\Rightarrow {\overline{\chi }}^2\left({\mathit{I}}_q,\mathit{M}{\mathcal{R}}_{*}\right)$$ (3.3)

where q, M and ${\mathcal{R}}_{*}$ were defined in the context of Theorem 2.1.

Theorem 3.1 shows that Theorem 2.1 can be extended to the case where Σ is unknown. Note that if λ_q, the value of the smallest nonzero eigenvalue of Σ, were known then Theorem 3.1 would hold with any fixed ε < λ_q. Similarly if q were known then simply set the smallest p − q eigenvalues to be zero. However in practice q and λ_q are not known. Hence we truncate the estimated eigenvalues by a small number ε_n. The stated conditions guarantee that with probability one rank(Σ_n (ε_n)) = rank(Σ) for large enough n. This rank condition ensures the convergence of the Moore–Penrose inverse and consequently the convergence of the associated quadratic forms (Andrews, 1987). An important feature of the proposed test is that it is well defined whether Σ is, or is not, singular, i.e., it adapts to the rank of Σ. Note that if ${\mathbf{\Sigma }}_n^{+}\stackrel{p}{\to }{\mathbf{\Sigma }}^{+}$ then the conclusion of Theorem 3.1 holds with ε_n = 0. However, there are many situations (cf., Cragg and Donald 1996) in which rank(Σ_n) = p for all n even though rank(Σ) < p. Obviously the proposed method will be useful in such a setting. We note that in most applications $b_n=\sqrt{n}$ so Theorem 2.1 is valid for any sequence ε_n = n^δ with δ < −1/2. For example we may choose ε_n ∝ 1/n^1/3. By choosing ε_n → 0 and b_nε_n → ∞ we are ensuring that in large samples we are not setting a nonzero eigenvalue to zero which means that with very high probability the rank of Σ will not be overestimated. A discussion of how to choose ε_n in practice is deferred to Section 3.3.

3.2. Winsorized tests

One may consider other forms of regularization of nearly singular variance matrices. One potential alternative to trimming is winsorizing as defined below.

Definition 3.2

Let ε > 0. We refer to the matrix Σ_W (ε) = EΛ_W (ε)E^T, where Λ_W (ε), which is the diagonal matrix obtained from Λ by replacing any element of Λ which is smaller than ε by Λ, as the Λ–winsorized variance matrix. Its inverse ${\mathit{E}}^T{\mathbf{\Lambda }}_W^{-1}(\epsilon )\mathit{E}$ is referred to as the ε–winsorized inverse of Σ and denoted by Σ^†(ε).

We note that winsorizing is closely related to ridging. However, in contrast with ridging only the small eigenvalues are modified. In fact the winsorized inverse shrinks 1/λ_i to a fixed constant 1/ε and is therefore of full rank. Both trimming and winsorizing can be obtained as special cases of ε–regularized inverses (Dufour and Valery, 2015).

Definition 3.3

Fix ε ≥ 0, the ε–regularized inverse of Σ is Σ_υ (ε) = E^TΛ _υ (ε)E where Λ_υ (ε) = diag(υ (λ₁, ε), …, υ (λ_p, ε)) and υ (λ, ε) is a nonnegative bounded function satisfying υ (λ, ε) = 1/λ when λ ≥ ε. The value of υ (λ, ε) when λ < ε is determined by the analyst. The function υ is referred to as a variance regularizing function.

For example, if υ (λ, ε) = 0 when λ < ε and λ_j < ε when j ≥ q + 1 theΛ _υ (ε) = diag(1/λ₁, …, 1/λ_q,0_p–q) where 0_p–q is a vector of 0′s with length p–q. It is clear that with this choice Λ_υ (ε), the regularized inverse, coincides with the trimmed inverse. Another natural choice is υ (λ, ε) = 1/λ when λ ≥ ε and υ (λ, ε) = 1/ε when λ ≤ λ which leads to Λ_υ (ε) = diag(1/λ₁, …, 1/λ_q, ε⁻¹1_p–q) in which case the regularized inverse coincides with the winsorized inverse. Since for λ ≤ ε we have 0 ≤ υ (λ, ε) ≤ ε it follows that all variance regularization functions interpolate between trimming and winsorizing, the two possibilities addressed in this paper.

Before presenting our next result we require the following notation. Let Y be ${\mathcal{N}}_p(\mathbf{0},\mathbf{\Sigma })$, define ${\overline{v}}^2(\mathbf{\Sigma },\mathcal{R})$ to be

$${\overline{v}}^2(\mathbf{\Sigma },\mathcal{R})={\mathit{Y}}^T\mathit{Y}-\min \left\{{(\mathit{Y}-\mathit{\theta })}^T(\mathit{Y}-\mathit{\theta }):\mathit{\theta }\in \mathcal{R}\right\}.$$

Note that the quadratic forms appearing in ${\overline{v}}^2(\mathbf{\Sigma },\mathcal{R})$ are the unweighted versions of those in ${\overline{\chi }}^2(\mathbf{\Sigma },\mathcal{R})$. We can now modify the statistic T_n (ε) by replacing ${\mathbf{\Sigma }}_n^{+}(\epsilon )$ with ${\mathbf{\Sigma }}_n^{†}(\epsilon )$ to obtain the winsorized constrained test statistic

$$W_n\left({\epsilon }_n\right)=\min \left\{n{\left({\mathit{S}}_n-\mathit{\theta }\right)}^T{\mathbf{\Sigma }}_n^{†}\left({\epsilon }_n\right)\left({\mathit{S}}_n-\mathit{\theta }\right):\mathit{\theta }\in {\mathcal{L}}_{*}^n\right\}-\min \left\{n{\left({\mathit{S}}_n-\mathit{\theta }\right)}^T{\mathbf{\Sigma }}_n^{†}\left({\epsilon }_n\right)\left({\mathit{S}}_n-\mathit{\theta }\right):\mathit{\theta }\in {\mathcal{C}}_{*}^n\right\},$$ (3.4)

where, as before, W_n = W_n (ε_n) when ε_n → 0. The limiting distribution of W_n and W_n (ε) are given in the following Theorem which applies to general variance regularizing functions. The matrices ${\mathit{M}}_1\in {ℝ}^{r\times p}$, ${\mathit{M}}_2\in {ℝ}^{(q-r)\times p}$ and ${\mathit{V}}_{q-r}\in {ℝ}^{(q-r)\times (q-r)}$ appearing in statement of the theorem below are defined in the proof Lemma S1.1 in the supplementary materials.

Theorem 3.2

Suppose that the assumptions given in Theorem 3.1 hold. Then

$$W_n\Rightarrow {\overline{\chi }}^2\left({\mathit{I}}_q,\mathit{M}{\mathcal{R}}_{*}\right),$$

with M as defined earlier. Furthermore, let r be the number of eigenvalues of Σ which are larger than ε. Then as n → ∞

$$W_n(\epsilon )\Rightarrow {\overline{\chi }}^2\left({\mathit{I}}_r,{\mathit{M}}_1{\mathcal{R}}_{*}\right)\oplus {\overline{v}}^2\left({\mathit{V}}_{q-r},{\mathit{M}}_2{\mathcal{R}}_{*}\right)$$ (3.5)

where ⊕ denotes the sum of independent RVs.

Theorem 3.2 is a generalization of Theorems 2.1 and 3.1. First, it shows that W_n and T_n have the same limiting distribution. This result may seem a bit surprising since ${\mathbf{\Sigma }}_n^{†}\left({\epsilon }_n\right)\to {\mathbf{\Sigma }}^{†}(0)$ and ${\mathbf{\Sigma }}_n^{+}\left({\epsilon }_n\right)\to {\mathbf{\Sigma }}^{+}$ but Σ⁺ ≠ Σ^† (0) whenever υ (0, 0) is nonzero. We also note that if ε is “sufficiently small”, i.e., if with high probability ε is smaller than λ_q,n, the q^th eigenvalue of Σ_n, then T_n (ε) and W_n (ε) coincide. If, however, ε is “too large” then instead of the limit (3.3) we obtain the limit (3.5). It is also clear that in this case $T_n(\epsilon )\Rightarrow {\overline{\chi }}^2\left({\mathit{I}}_r,{\mathit{M}}_1{\mathcal{R}}_{*}\right)$ as n → ∞, where the matrix M₁ is defined in the proof of Lemma S1.1 in the supplementary materials. The relative merits of these statistic are investigated below.

Proposition 3.1

Under the stated conditions the tests T_n, W_n and W_n (ε) are consistent. The test T_n (ε) is consistent if and only if ${\mathit{M}}_1\mathit{\theta }\in {\mathit{M}}_1{\mathcal{R}}_{*}\backslash \left\{\mathbf{0}\right\}$.

Proposition 3.1 shows that a test based on T_n (ε) may not be consistent for some configurations in the alternative. Nevertheless, for other configurations it may be more powerful than W_n (ε), W_n or T_n. Hence no test dominates the others as demonstrated in the simple example below.

Example 3.1

Assume that Y₁, Y₂, …, are ${\mathcal{N}}_3(\mathit{\theta },\mathbf{\Sigma })$ with $\mathbf{\Sigma }=\mathrm{diag}\left({\sigma }_1^2,{\sigma }_2^2,{\sigma }_3^2\right)$ where ${\sigma }_1^2>{\sigma }_2^2>{\sigma }_3^2=0$. It is further assumed that Σ is not known in advance. Consider testing H₀ : θ= 0 versus $H_1:\mathit{\theta }\in {ℝ}_{+}^3\backslash \left\{\mathbf{0}\right\}$. Let ${\overline{\mathit{Y}}}_n=n^{-1}{\sum }_1^n{\mathit{Y}}_i$ and ${\mathbf{\Sigma }}_n=n^{-1}{\sum }_1^n\left({\mathit{Y}}_i-{\overline{\mathit{Y}}}_n\right){\left({\mathit{Y}}_i-{\overline{\mathit{Y}}}_n\right)}^T$. Choose ε such that ${\sigma }_1^2>\epsilon >{\sigma }_2^2$. It can be verified that for large n

$$T_n=\max (0,Z_1+\sqrt{n}\frac{{\theta }_1}{{\sigma }_1}{)}^2+\max (0,Z_2+\sqrt{n}\frac{{\theta }_2}{{\sigma }_2}{)}^2+o_p(1)$$

$$W_n=\max (0,Z_1+\sqrt{n}\frac{{\theta }_1}{{\sigma }_1}{)}^2+\max (0,Z_2+\sqrt{n}\frac{{\theta }_2}{{\sigma }_2}{)}^2+o_p(1)$$

$$T_n(\epsilon )=\max (0,Z_1+\sqrt{n}\frac{{\theta }_1}{{\sigma }_1}{)}^2+o_p(1)$$

$$W_n(\epsilon )=\max (0,Z_1+\sqrt{n}\frac{{\theta }_1}{{\sigma }_1}{)}^2+\frac{{\sigma }_2^2}{\epsilon }\max (0,Z_2+\sqrt{n}\frac{{\theta }_2}{{\sigma }_2}{)}^2+o_p(1)$$

where Z₁, Z₂ are independent $\mathcal{N}(0,1)$ RVs. It is clear that the distribution of T_n (ε) is the same for all (θ_1, θ₂) ∈ {(0, c) : c ≥ 0}. It follows that T_n (ε) can not detect if θ₂ > 0. However if (θ₁, θ₂) = (c, 0) and c > 0 then T_n (ε) will be more powerful than the other tests. This will also often be the case when θ₁ is large and θ₂ is small. The relative merit of W_n (ε) compared with W_n depends on the value of both ${\sigma }_2^2/\epsilon$ and (θ₁, θ₂) and is more difficult to assess. Note that T_n (and W_n) is actually the test one would use if it were known in advance that Σ is diagonal with ${\sigma }_1^2>0$, ${\sigma }_2^2>0$ and ${\sigma }_3^2=0$.

3.3. Choosing ε and two supremum statistics

The statistics (3.2) and (3.4) are functions of ε. We now describe a few approaches for choosing ε when using (3.2) or (3.4). First, in some settings there may be prior knowledge regarding the rank of Σ or the size of its smallest eigenvalue. Such information, when available, should of course guide our choice. In general Σ is not known and therefore it seems that an automated data driven procedure should be developed. A natural approach, motivated by numerical stability considerations, is to truncate or winsorize the small eigenvalues until the modified condition number of Σ_n, i.e., the ratio of the largest to the smallest nonzero eigenvalue, is bounded by some constant. More concretely let λ_1,n ≥ · · · ≥ λ_p,n denote the eigenvalues of the Σ_n. Let

$$\kappa =\max \{i:\frac{{\lambda }_{1,n}}{{\lambda }_{i,n}}\le \mathrm{\Delta }\}$$

where Δ is a pre–specified constant (typically between 50–200), then choose ε_n = λ_k,n. This type of approach is well known and extensively discussed in the literature on numerical linear algebra where systems of linear equations with large condition numbers are solved (El Ghaoui 2002, Montgomery and Peck 2007, and Golub and Van Loan 2012). We remark that this method is a variation on the choice provided by Fan et al. (2012). However, their choice was inspired by multiple testing problems for correlated data where the objective was to approximate the population covariance by a matrix derived from a factor analytic model. Although numerical experiments show that this approach is flexible, easy to implement and results in a good power for the test statistic under consideration, we have developed a more general approach discussed below.

Recall that by Theorems 3.1 and 3.2 if ε_n → 0 and b_nε_n → ∞ then T_n and W_n converge to the same limit. Usually $b_n=\sqrt{n}$, therefore the Theorems will be satisfied if ε = C/n^1/k for some constant C and k > 2. A reasonable choice is k = 3 and C = λ_1,n, i.e., each eigenvalue is compared to largest estimated eigenvalue scaled by 1/n^1/3. Obviously the aforementioned choice of the constant C is arbitrary and one can compare to any other function of the eigenvalues such as the trace, the product of the eigenvalues, the sum/product of the top r eigenvalues and so forth. Although, in large samples, all such choices are equivalent there may be differences in small to moderate samples. Moreover, it is important to note that any scaling, 1/n^1/k, with k > 2 will result in the same limit. This is very different from many other statistical problems in which the order of the regularizing parameter is very important, see Cule et al. (2011), Bulmann (2013), Van de Geer et al. (2014) for ridge regression and lasso examples and Wand and Jones (1995) for kernel smoothing. Moreover, note that in all of the abovementioned papers the constant factor is also chosen in an ad–hoc manner.

Conclusion 3.1

Choosing ε in (3.2) and (3.4) is fundamentally different from the problem of choosing the value of a regularizing parameter in settings such as ridge regression or nonparametric regression. First, regardless of the value chosen for ε the resulting test always has the prescribed level, at least asymptotically. Consequently the choice of ε may only affect power. Our numerical experiments indicate that usually the power is only marginally influenced by the choice of ε. Moreover, and as indicated by Proposition 3.1 and Example 3.1, the power of W_n(ε) is much less influenced by the choice of ε compared with T_n(ε). It is also important to realize that there does not exist an optimal choice of ε as the best ε is a function of the unknown value of θ. In contrast, the choice of the ridging parameter affects the performance of both estimators and tests, e.g., Vinod and Ullah (1981) as well as the abovementioned references. The same holds for nonparametric regression where the regularizing parameter is chosen to balance the bias and variance. Such a balancing act is not required here.

Nevertheless, some users may be uncomfortable choosing a single value for ε. To accommodate such a preference we introduce two supremum statistics, based on (3.2) and (3.4), which depend on a range of ε values. Let 0 < ε₁ ≤ ε₂ and set I = [ε₁, ε₂]. Define,

$$T_n(I)=\underset{\epsilon \in I}{\sup }{T}_n(\epsilon )\text{and}{W}_n(I)=\underset{\epsilon \in I}{\sup }{W}_n(\epsilon ).$$ (3.6)

The statistics (3.6) are motivated by Davies (1987) and Andrews and Ploberger (1994) where tests are developed for the case where a nuisance parameter is present only under the alternative. Here ε varies over the interval I and plays the role of the nuisance parameter.

In the following we derive the limiting distributions of the statistics (3.6). First fix ε > 0. Then by equation (3.2) and from the proof of Theorem 3.1, given in the supplementary materials, it follows that

$$T_n(\epsilon )\Rightarrow T(\epsilon )={\mathit{X}}^T{\mathit{I}}_{r(\epsilon )}\mathit{X}-\min \left\{{(\mathit{X}-\mathit{\eta })}^T{\mathit{I}}_{r(\epsilon )}(\mathit{X}-\mathit{\eta }):\mathit{\eta }\in {\mathit{M}}^{\epsilon }{\mathcal{R}}_{*}^{\epsilon }\right\},$$

where X is a ${\mathcal{N}}_q(\mathbf{0},\mathit{I})\mathrm{RV}$, $r(\epsilon )={\sum }_{i=1}^p{\mathbb{I}}_{\left\{{\lambda }_i\ge \epsilon \right\}}$ is the number of eigenvalues larger than ε, I_r(ε) = diag(1_r(ε), 0_q–r(ε)), M^ε is the r(ε) × p matrix obtained by dropping the last p–r(ε) rows of the matrix (Λ⁺(ε))^1/2E and ${\mathcal{R}}_{*}^{\epsilon }=\mathcal{C}\cap {\mathcal{L}}^{\perp }\cap \mathcal{S}(\mathbf{\Sigma }(\epsilon ))$. Conditioning on X we note that T (ε) is a function of ε only through r(ε), which is a step function. This implies that

$$\{T(\epsilon ):\epsilon \in I\}\subset \left\{T\left({\lambda }_i\right):i=j,\dots ,k\right\}$$ (3.7)

where j = min {i : λ_i ≥ ε₂} and k = min {i : λ_i ≥ ε₁}, i.e., T (ε) assumes only a finite number of values. Finally we note that conditionally T (ε) is a decreasing function of ε. Therefore it follows by Theorem 1.A.6 in Shaked and Shanthikumar (2007) that unconditionally we have $T\left({\epsilon }_2\right){⪯}_{st}T\left({\epsilon }_1\right)$ whenever ε₁ ≥ ε₂ where ${⪯}_{st}$ denotes the usual stochastic order.

Similarly, by Theorem 3.2

$$W_n(\epsilon )\Rightarrow W(\epsilon )=W_1(\epsilon )+W_2(\epsilon )$$

where

$$W_1(\epsilon )={\mathit{X}}_1^T{\mathit{I}}_{r(\epsilon )}{\mathit{X}}_1-\min \{{\left({\mathit{X}}_1-\mathit{\eta }\right)}^T{\mathit{I}}_{r(\epsilon )}\left({\mathit{X}}_1-\mathit{\eta }\right):\mathit{\eta }\in {\mathit{M}}_1^{\epsilon }{\mathcal{R}}_{*}\},$$

$$W_2(\epsilon )={\mathit{X}}_2^T{\mathit{I}}_{q-r(\epsilon )}{\mathit{X}}_2-\min \{{\left({\mathit{X}}_2-\mathit{\eta }\right)}^T{\mathit{I}}_{q-r(\epsilon )}\left({\mathit{X}}_2-\mathit{\eta }\right):\mathit{\eta }\in {\mathit{M}}_2^{\epsilon }{\mathcal{R}}_{*}\},$$

X₁ and X₂ are independent ${\mathcal{N}}_{r(\epsilon )}(\mathbf{0},\mathit{I})$ and ${\mathcal{N}}_{q-r(\epsilon )}\left(\mathbf{0},{\mathit{V}}^{\epsilon }\right)$ and V^ε = diag(λ_r(ε)+1/ε, …, λ_q/ε). The matrices ${\mathit{M}}_1^{\epsilon }$, ${\mathit{M}}_2^{\epsilon }$ are defined as in Theorem 3.2 with r = r(ε). To gain some insight into the behavior of W(ε) consider the setup in Example 3.1, where p = 3, q = 2, $\mathbf{\Sigma }=\mathrm{diag}\left({\sigma }_1^2,{\sigma }_2^2,{\sigma }_3^2\right)$ with ${\sigma }_1^2>{\sigma }_2^2>{\sigma }_3^2=0$, for which a direct calculation shows that

$$W(\epsilon )=\psi \left(\epsilon ,{\sigma }_1^2\right)\max {\left(0,X_1\right)}^2+\psi \left(\epsilon ,{\sigma }_2^2\right)\max {\left(0,X_2\right)}^2$$

where X = (X₁, X₂)^T is a ${\mathcal{N}}_2(\mathbf{0},\mathit{I})$ RV and ψ(ε, σ²) = 1 if σ² ≥ ε and σ²/ε if σ² < ε. Thus conditional on X, W (ε) is a decreasing, continuous function of ε. Furthermore, arguing as before, it can be shown that unconditionally $W\left({\epsilon }_2\right){⪯}_{st}W\left({\epsilon }_1\right)$. To summarize:

Theorem 3.3

Suppose that the assumptions given in Theorems 3.1 and 3.2 hold. Then as n → ∞ we have

$$T_n(I)\Rightarrow T\left({\epsilon }_1\right)\mathit{and}{W}_n(I)\Rightarrow W\left({\epsilon }_1\right).$$

Theorem 3.3 shows that the theoretical critical values of the statistics T_n (I) and W_n (I) should be computed at ε₁. However, in finite samples T_n (I) and W_n (I) may not be maximized at ε₁ even under the null. In practice the interval I can be chosen using the estimated eigenvalues of Σ_n. We propose choosing I = [λ_p,n, λ_1,n], i.e., maximizing (3.2) and (3.4) over the full spectrum of Σ_n. Note that choosing values of ε₁ smaller than λ_p,n will not yield numerically different results, whereas choosing ε₂ larger than λ_1,n will result in a loss of power. Our numerical studies, as well Theorem 3.3, indicate that the type I error is always maintained. Moreover as indicated by our simulations the power of the resulting tests is comparable to the power obtained by choosing the best value for ε when its value is known in advance. Thus our approach completely eliminates the need to choose a value for ε while hardly compromising the power of the test.

Finally, we emphasize that the statistics (3.2) and (3.4) as well as (3.6) converge to the same limit if the conditions in Theorem 3.1 are satisfied. The consistency of tests based on T_n (I) and W_n (I) follows immediately from Proposition 3.1.

3.4. Implementation

Finally a few remarks regarding implementation. Suppose first, that the statistic S_n satisfies (1.1) and that Σ is known. If so, find Σ⁺ the spectral cut–off Moore–Penrose inverse of Σ, as described in Section 2, and compute the statistic T_n given by (2.1) using rank reduced quadratic programming. Denote its observed value by t_n. To calculate the significance level:

Step 1: Simulate a sample of size n from a multivariate normal distribution with mean vector 0 and variance matrix Σ.

Step 2: Compute ${\widehat{\mathit{\theta }}}_i$ the sample mean for the i^th sample.

Step 3: Compute the test statistic T_n,i in the same manner, using ${\widehat{\mathit{\theta }}}_i$ in place of ${\widehat{\mathit{\theta }}}_i$.

Repeat Steps 1–3 B times, where B is large enough to ensure that the null distribution is estimated sufficiently well. In our examples we used B = 10⁵. We then approximate the p–value by

$$\frac{1}{B}\sum _{i=1}^B{\mathbb{I}}_{\left(T_{n,i}>t_n\right)}.$$ (3.8)

Modifications of the procedure above are necessary when the variance matrix Σ is unknown. We will describe the procedure for the supremum statistic W_n (I); analogous procedures are used for (3.2), (3.4) and T_n (I). The first step is to calculate Σ_n, a consistent estimator of Σ. Given Σ_n an interval I of values for ε is set forth as described in Section 3.3. The observed value of the statistic is calculated using (3.4) and (3.6) and denoted ω_n(I). The significance level is calculated by repeating steps 1–3 above, where: in Step 1 we simulate from a multivariate normal distribution with mean vector 0 and variance matrix Σ_n; Step 2 is exactly as described above and in Step 3 we compute W_n,i(I), where at each iteration we reestimate Σ. Finally the p–value is calculated as in (3.8) with T_n,i replaced by W_n,i(I) and t_n replaced by w_n(I). Our approach is quite standard and falls within the framework of Monte–Carlo tests (cf. Dufour 2006, MacKinnon 2009), where a plug–in estimator for the nuisance parameter is used when constructing the test statistic. Of course this approach relies on large sample theory. It is clear from our simulations that it works well in this application. These procedures described above were programmed in MATLAB.

4. Simulations

We conducted a wide range of simulation studies to compare the proposed test statistics. More specifically, we provide two different simulation settings in which the type I errors and powers of the proposed statistics are compared. We focus on the statistics T_n = T_n (0), T_n(I) and W_n(I), although results for T_n(ε) and W_n(ε) for various values of ε are also reported but presented only in the Supplementary Materials. The first simulation setting is inspired by Example 3.1 and is shortly described whereas the second simulation setting is inspired by the Fibroid Growth Study (FGS). The FGS and the corresponding data analyzed in this paper are described in detail in Section 5.

Motivated by Example 3.1, we generated n observations from a ${\mathcal{N}}_p(\mathit{\theta },\mathbf{\Sigma })$ distribution, with either p = 3 or p = 8. For the case where p = 3 we chose

$${\mathbf{\Sigma }}_{3\times 3}=\left(\begin{array}{ccc}1.00& 0.25& 0\\ 0.25& 1.00& 0\\ 0& 0& 0\end{array}\right).$$

This variance matrix has eigenvalues 1.25, 0.75, and 0.00. Under the null hypothesis, the mean vector θ was taken to be the null vector. We considered three different patterns of alternative hypotheses, namely, the simple order, the orthant order, and the tree order. The values of θ under the corresponding alternatives is given in Table 1. For the case of p = 8 we chose

$${\mathbf{\Sigma }}_{8\times 8}=\left(\begin{array}{cc}\mathbf{\Omega }& {\mathbf{0}}_{4\times 4}\\ {\mathbf{0}}_{4\times 4}& \mathbf{\Psi }\end{array}\right),$$

where Ω = (1 – ρ)I₄ + ρJ₄, I₄ is the identify matrix, J₄ is the 4 × 4 matrix of all 1’s, and Ψ = {10⁻³, 10⁻⁵, 0, 0}. For this simulation, we set ρ = 0.25. The resulting variance matrix has eigenvalues 1.75, 0.75, 0.75, 0.75, 10⁻³, 10⁻⁵, 0, and 0. Thus in this example, two eigenvalues are identically zero and at least one is nearly zero. Once again we considered patterns of mean vectors satisfying either the simple order, the orthant order, or the tree order alternative. The values of θ under the corresponding alternatives is given in Table 2.

Table 1:

Patterns of the mean vector under the alternative hypothesis for the case of p = 3.

		Value of θi
	i =	1	2	3
	Simple	0.00	0.25	0.00
Design A	Orthant	0.50	0.50	0.00
	Tree	0.00	0.50	0.00
	Simple	0.00	0.25	0.50
Design B	Orthant	0.50	0.50	0.50
	Tree	0.00	0.50	0.50

Table 2:

Patterns of the mean vector under the alternative hypothesis for the case of p = 8.

		Value of θi
	i =	1	2	3	4	5	6	7	8
	Simple	0.00	0.01	0.03	0.04	0.06	0.07	0.00	0.00
Design A	Orthant	0.20	0.20	0.20	0.00	0.00	0.00	0.00	0.00
	Tree	0.00	0.10	0.10	0.10	0.00	0.00	0.00	0.00
	Simple	0.00	0.01	0.03	0.04	0.06	0.07	0.09	0.10
Design B	Orthant	0.20	0.20	0.20	0.00	0.00	0.00	0.00	0.20
	Tree	0.00	0.10	0.10	0.10	0.00	0.00	0.00	0.10

For both p = 3 and p = 8, the interval of values I for ε were taken to be all non–zero eigenvalues of Σ. For the case Σ known we used 10, 000 replications to simulate the sampling distribution of all three test statistics under the null hypothesis and the type I error rates and powers were estimated using 5, 000 replications. In the case of Σ unknown, which is more realistic, the test statistic was calculated using the sample covariance matrix. Replications to simulate the sampling distribution were drawn from a multivariate Normal distribution using the true Σ, but the test statistics for these replications were computed using the sample covariance matrix.

All three tests controlled the type I errors at their nominal rate of 0.05 (see Tables S3 and S5 in the Supplementary Materials, Section S2). This is true whether Σ is known or unknown. Note that the results for known Σ are of lesser importance and provided as benchmark. The discussion below focuses on the case where Σ is unknown. Results for powers for p = 3 are summarized in Figure 1, and the powers for p = 8 are summarized in Figure 2. Complete results are provided in Tables S4 and S6 in the Supplementary Materials. For all patterns considered the power of W_n(I) generally exceeded the power of both T_n(I) and T_n(0), except for the simple order for p = 3. A bit of thought reveals why. Note that in the case of the simple order T_n(0) actually tests for θ₁ < θ₂ in ${ℝ}^2$, whereas W_n(I) tests whether {θ₁ < θ₂} ∪ {θ₂ < θ₃} on ${ℝ}^3$. However, in Design A θ₃ = 0, so actually under the alternative θ₁ < θ₂ > θ₃ which is an umbrella order, not a simple order. Thus the power of the test, designed for the simple order, is reduced. Moreover, if we test for the correct alternative, i.e., the umbrella order, then W_n(I) is again the most powerful test. For example when n = 10 the powers are 0.0930 for T_n(I), 0.1448 for T_n(0) and 0.1866 for W_n(I). The gain in power associated with W_n(I) is often substantial and when W_n(I) is not the most powerful it typically loses only by a small margin. Consequently, we recommend that investigators use the statistic W_n(I). Furthernote that under Design B, the gains are even more substantial for both p = 3 and p = 8. See Figures 3 and 4 as well as Tables S8, S10, and S11 in the Supplementary Materials. These Figure indicate that the W_n(I) statistic is more robust with respect to singularity of Σ and therefore also more powerful.

Figure 1:

Powers for W_n(I) (triangle, dotted lines), T_n(I) (diamonds, dashed lines), and T_n(0) (circles, solid lines) for simulation 1, Design A, the case where p = 3.

Figure 2:

Powers for W_n(I) (triangle, dotted lines), T_n(I) (diamonds, dashed lines), and T_n(0) (circles, solid lines) for simulation 1, Design A, the case where p = 8.

Figure 3:

Powers for W_n(I) (triangle, dotted lines), T_n(I) (diamonds, dashed lines), and T_n(0) (circles, solid lines) for simulation 1, Design B, the case where p = 3.

Figure 4:

Powers for W_n(I) (triangle, dotted lines), T_n(I) (diamonds, dashed lines), and T_n(0) (circles, solid lines) for simulation 1, Design B, the case where p = 8.

5. Illustration: changes in gene expression by the tumor size in the NIEHS Fibroid Growth Study

Uterine leiomyoma (also called uterine fibroids) are benign hormonally mediated smooth muscle tumors commonly found in premenopausal women. According to some estimates at least 70% of white women in the US have fibroids and the estimates are even higher for black women. Furthermore, the total annual cost associated with these tumors in the US is over 30 billion dollars. Despite such high prevalence rates and associated costs, these tumors are not very well characterized. The NIEHS Fibroid Growth Study (FGS) is one of the largest studies, involving 72 premenopausal women, where the growth patterns of fibroids were investigated (Peddada et al., 2008). Tissue samples obtained from women in the FGS who underwent either myomectomy or hysterectomy provide a great opportunity for researchers to investigate the molecular characteristics of these benign tumors. In this section we compare the mean expression of two sets of important genes using the proposed methodology. Specifically, we consider a subset of genes belonging to the Interleukin–1 (IL–1) signaling pathway which are known to be involved in the initial stages of tumor formation (Dunne et al., 2003) and a subset of genes called collagens which are well–known to be associated with tumor development and growth (Davis et al., 2013, Leppert et al., 2014). Following Grandhi et al. (2016) we shall consider four groups: (1) normal myometrium; (2) small tumors (between 0.08 – 5.70 cm³); (3) medium tumors (between 9.0–132.00 cm³); and large tumors (between 240–2016 cm³). The respective sample sizes were, n₁ = 8, n₂ = 14, n₃ = 25 and n₄ = 13.

Let Y_ij denote the m dimensional gene expression vector of the j^th individual in the i^th group. Let ${\overline{\mathit{Y}}}_{i\cdot }=n_i^{-1}{\sum }_j{\mathit{Y}}_{ij}$, i = 1, …, 4 denote the group means. Let ${\mathit{\theta }}_i=\mathbb{E}\left({\mathit{Y}}_{ij}\right)$ be the mean gene expression level in group i. We also assume that $\mathbb{V}ar\left({\mathit{Y}}_{ij}\right)=\mathbf{\Psi }$ for all i and j, i.e., the variance are homogeneous across groups. Define,

$${\mathit{Z}}_n^T=\left(\sqrt{n_1}\left({\overline{\mathit{Y}}}_{1\cdot }-{\mathit{\theta }}_1\right),\sqrt{n_2}\left({\overline{\mathit{Y}}}_{2\cdot }-{\mathit{\theta }}_2\right),\sqrt{n_3}\left({\overline{\mathit{Y}}}_{3\cdot }-{\mathit{\theta }}_3\right),\sqrt{n_4}\left({\overline{\mathit{Y}}}_{4\cdot }-{\mathit{\theta }}_4\right)\right).$$

Clearly $\mathbb{E}\left({\mathit{Z}}_n\right)=0$ and $\mathbf{\Sigma }=\mathbb{V}ar\left({\mathit{Z}}_n\right)=\mathit{I}\otimes \mathbf{\Psi }$, where Ψ is m × m and I is the 4 × 4 identity matrix. Hence is a p × p (p = 4m) block diagonal matrix with blocks Ψ repeated 4 times. We shall assume that ${\mathit{Z}}_n\sim {\mathcal{N}}_p(\mathbf{0},\mathbf{\Sigma })$.

An important component of growth and development of fibroids is angiogenesis or vascularization, the formation of blood vessels. According to Grandhi et al. (2016), the Interleukin-1 (IL–1) signaling pathway is involved in a cascade of events including inflammatory response and production of prostaglandins that trigger the expression of growth factors involved in angiogenesis. Therefore in this paper we investigated the relationship between tumor size and gene expression patterns of m = 25 genes belonging to the IL–1 signaling pathway. The names of the 25 genes along with the log–transformed (to base 2) mean gene expression data, and pooled variance matrix, are provided in Supplementary Tables S12 and S13, respectively. A priori we do not know if the 25 genes considered in this analysis are positively or negatively associated with tumor size. Hence we tested the null hypothesis H₀ : θ₁ = θ₂ = θ₃ = θ₄ against the alternative $H_{a_1}\cup H_{a_2}$ where $H_{a_1}:{\mathit{\theta }}_1\le {\mathit{\theta }}_2\le {\mathit{\theta }}_3\le {\mathit{\theta }}_4$ and $H_{a_2}:{\mathit{\theta }}_1\ge {\mathit{\theta }}_2\ge {\mathit{\theta }}_3\ge {\mathit{\theta }}_4$. Stacking θ = (θ₁, θ₂, θ₃, θ₄)^T and choosing a suitable contrast matrix C we restate the above hypotheses as H₀ : Cθ = 0, versus H₁ : {Cθ ≥ 0} ∪ {Cθ ≥ 0}. As often done (Peddada et al. 2003), we tested this union-intersection hypotheses by testing for each alternative hypothesis separately and then applying Bonferroni corrections to the resulting p–values. Thus suppose p₁ is the p–value associated with testing H₀ versus $H_{a_1}$ and p₂ is the p–value associated with testing H₀ versus $H_{a_1}$, then the Bonferroni corrected p–values are min(1, 2p_i) i = 1, 2.

5.1. Data–driven simulation study

A simulation study motivated by the data obtained from the FGS was carried out. Data from four p–variate normal distributions was generated. A test for H₀ : θ₁ = θ₂ = θ₃ = θ₄ against the alternative H₁ : θ₁ ≤ θ₂ ≤ θ₃ ≤ θ₄ was conducted as described above. By stacking the random vectors from the four groups, the data can be represented by a single 4m × 1 normal vector with a 4m × 1 population mean vector and 4m × 4m variance matrix of the form Σ = I ⊗ Ψ, where I is a 4 × 4 identity matrix, Ψ is a m × m matrix and ⊗ denotes the usual Kronecker product of matrices. Thus in terms of the previous notations p = 4m and m = 25. Our choice of parameters for the multivariate normal distribution were inspired by the real data in the IL–1 signaling pathway in FGS. Let θ denote the pooled sample mean vector and let Ψ denote the pooled sample covariance matrix of the 25 genes in the IL–1 signaling pathway from four groups. Out of the 25 genes, we arbitrarily chose five of the genes (i.e. components 4, 8, 12, 16, and 20 of the mean vector) to satisfy the alternative hypothesis and the remaining genes (or components) satisfy the null hypothesis. For these five non-null genes, the elements of θ were set to increase by 1% from group to group. For example, the means of the 4^th gene were θ₁₄, 1.01θ₂₄, 1.02θ₃₄, 1.03θ₄₄, for groups 1, 2, 3, and 4, respectively. See Supplementary Tables S12 and S13 for the values of the mean vectors, and Ψ.

Our goal was to investigate the effect of the degree of singularity, as measured by the condition number of the matrix Ψ, on the performance of the proposed tests. Recall that we may write the spectral decomposition of Ψ as Ψ = EΛE^T where Λ is a diagonal matrix of the eigenvalues. Denote by Λ₂ the eigenvalues observed in the FGS. In addition we constructed the matrices Λ₁ an Λ₃ defined by Λ₁ = s × W₁ and Λ₃ = s × W₃ where s is the sum of the eigenvalues of Λ₂ and W_j = diag(ω_j), j = 1, 2 are diagonal weight matrices with decreasing weights (i.e., ${\mathit{w}}_{ji}\ge {\mathit{w}}_{j{i}^{\prime }}$ when i ≤ i′), which preserve the sum of the eigenvalues while modifying the condition number, see Table S16 in the Supplementary text. Specifically Λ₁ has a condition number of Δ = 24 and Λ₃ has a condition number of Δ = 1301.

The performance of the proposed tests was evaluated assuming the variance was unknown. The results are displayed in Figure 5, and in the Supplementary Materials, Section S3. While T_n(I) occasionally achieves higher power than W_n(I), this appears to be limited to the extreme cases when n = 10 and the condition number is very high. We speculate that T_n(I) outperforms W_n(I) when n = 10 because in this setting with only 40 observations, it is di¢ cult to precisely estimate the variances and covariances and T_n(I) which trims the eigenvalues is able to reduce the amount of noise while W_n(I) does not. Nevertheless, considered overall, W_n(I) still appears to be the preferable statistic: it consistently has equal or higher power than T_n and, outside of the extreme cases, also outperforms T_n(I).

Figure 5:

Powers for W_n(I) (triangle, dotted lines), T_n(I) (diamonds, dashed lines), and T_n(0) (circles, solid lines) for simulation design 2.

5.2. Data analysis

The variance matrix has 25 eigenvalues with a condition number of 557. For illustration purposes we conducted the tests T_n(I), W_n(I), and T_n(0). The interval I, for computing W_n(I) and T_n(I), was selected to cover the full range of eigenvalues of Ψ. Thus the set I included the 1^st, 5^th, 10^th, 15^th, and 25^th eigenvalues. All tests were carried out assuming that Σ was unknown. According to our analysis, after performing Bonferroni corrections, T_n(I) rejected the null hypothesis in favor of an increasing trend, but neither W_n(I) not T_n(0) did so. The respective p–values were 0.007, 1, and 1. Similarly, T_n(I) declared a significant decreasing trend, while W_n(I) and T_n(0) did not, with Bonferroni corrected p–values of 0.007, 0.36, and 0.64. However, it should be noted that the null distribution of T_n(I) was highly skewed. Of the 10,000 replications used to simulate the null distribution of T_n(I) for the increasing trend, 9965 of them resulted in a test statistic of exactly 0. The degree of singularity in the IL–1 signaling pathway data may render T_n(I) an untrustworthy test.

We emphasize that this is the first paper to assess the association between tumor size and gene expression in this context. There are two possible explanations for not detecting a significant increasing or a significant decreasing trend in the gene expression of the vector of 25 IL–1 pathway genes considered here. The first is that failing to reject a null hypothesis does not imply the null is true but may just be an indication that the study may be underpowered for discovering the multivariate trends considered in this paper. The second is that some genes are positively associated while others are negatively associated (and some may not be associated at all). To discern if the latter is the case, we performed a univariate trend analyses for each gene separately using ORIOGEN (Peddada et al., 2003, Peddada et al., 2005), a bootstrap based univariate order restricted inference method to detect patterns in an ordered gene expression data. After controlling the false discover rate at 5% we discovered that 4 out of 25 genes had a highly significant increasing trend whereas all remaining 21 genes had a highly significant decreasing trend. Results of these ORIOGEN based analyses are provided in the Supplementary text (Table S16). Due to these conflicting trends it is not surprising that the multivariate trend test did not find all 25 genes to have significant increasing trend or all to have a significant decreasing trend. Moreover, Grandhi et al. (2016) discovered that the genes in the IL–1 pathway were enriched only in the small tumors and hence are potentially involved in tumor genesis and initiation. Therefore we performed an additional post–hoc analysis using ORIOGEN to investigate if there were significant differences in the gene expression between small and medium tumors and between small and large tumors. Consistent with Grandhi et al. (2016) we discovered that none of these 25 genes were differentially expressed in these pairwise comparisons (at FDR = 0.05) although we found all of them to be differentially expressed between the normal myometrium and the small tumors.

6. Summary and discussion

In this article we introduced a general framework and concrete tools for testing hypotheses under linear inequality constraints when the variance matrix is potentially singular. The new methodology extends the existing literature in several directions and since it adapts to the true rank of Σ it can handle singularity as well as nearly singularity of variance matrices in a unified way. Adding to the novelty, the ideas of trimming and winsorizing are extended to the present context of singular and nearly singular variance matrices. It is well–known that these techniques were originally developed to deal with extreme observations in the data. Specifically we propose trimmed and winsorized tests (3.2) and (3.4) and two supremum tests (3.6) which are based on them. Although all tests are asymptotically equivalent there are substantial differences in power in finite sample. Interestingly, when comparing (3.2) and (3.4), we discovered that in most cases the test statistic based on the winsorized variance matrix has larger power than the test based on the trimmed variance matrix. The gain in power was often substantial. Recall that, under normality, winsorizing leads to asymptotically efficient estimators and is therefore preferred to trimming. Similarly we found that the statistic W_n(I) results in more powerful tests than T_n(I). Further we observe that the supremum tests (3.6), which do not require choosing ε are only minimally less powerful than the tests (3.2) and (3.4) assuming that the best value for ε is known. For these reasons we recommend using W_n(I).

The proposed methodology can be extended in several directions. As an example we discuss the regression setting. First note that in order to estimate a regression parameter it is often necessary to first invert the variance matrix Σ. This is not possible if Σ is singular. Hence additional assumptions on the data generating mechanism, beyond those given by (1.1), are required. Of course, if the matrix X^T X has very large but finite condition number, then the ordinary least squares estimator exists and is unbiased and a trimming or winsorizing test can be applied as discussed above. This holds true whenever the matrix $n^{-1}{\sum }_i{\mathit{x}}_i{\mathit{x}}_i^T$ converges to a non–singular matrix. However, if $n^{-1}{\sum }_i{\mathit{x}}_i{\mathit{x}}_i^T$ converges to a singular matrix, as in Knight (2008), then a different analysis is required. To fix ideas consider the linear regression Y_i = β^Tx_i + ε_i where i = 1, …, n. The least squares estimator is given by

$${\widehat{\mathit{\beta }}}_n=\mathit{\beta }+{\left({\mathit{X}}_n^T{\mathit{X}}_n\right)}^{-1}{\mathit{X}}_n^T{\epsilon }_n.$$

Note that if ε_i are independent $\mathcal{N}\left(0,{\sigma }^2\right)$ RVs then

$$c_n({\widehat{\mathit{\beta }}}_n-\mathit{\beta })$$ (6.1)

is distributed as ${\mathcal{N}}_p(\mathbf{0},{\sigma }^2{c}_n^2{({\mathit{X}}_n^T{\mathit{X}}_n)}^{-1})$ for all n. A limiting distribution exists provided $c_n^2{\left({\mathit{X}}_n^T{\mathit{X}}_n\right)}^{-1}$ converges. It is typically assumed that $n^{-1}{\left({\mathit{X}}_n^T{\mathit{X}}_n\right)}^{-1}$ converges to a matrix that is positive definite, in which case we obtain $c_n=\sqrt{n}$, the usual convergence rate. Consider now the case where as n → ∞ we have $\Vert {\mathit{x}}_n-\mathcal{S}\Vert \to 0$ where $\mathcal{S}\subset {ℝ}^p$ is a subspace, i.e., the observations are increasingly concentrated on a subspace. It follows that $n^{-1}{\sum }_{i=1}^n{\mathit{x}}_i{\mathit{x}}_i^T\to \mathit{S}$ where S is some singular matrix. We first exclude the possibility that ${\lim }_n{\sum }_i{\mathit{x}}_i{\mathit{x}}_i^T<\infty$ in which case ${\widehat{\mathit{\beta }}}_n$ is not consistent. It is further evident that a limit in (6.1) will exist if and only if $c_n^{-2}\left({\mathit{X}}_n^T{\mathit{X}}_n\right)$ converges to some positive definite matrix and c_n is a sequence of constants increasing to infinity. By letting $\Vert {\mathit{x}}_n-\mathcal{S}\Vert \to 0$ at various rates we can obtain different normalizing constants c_n. It follows that our methodology will apply here if we replace $\sqrt{n}$ with c_n in (1.1) and proceeding as before.

In this paper the focus had been on the case where Σ is not a function of the unknown mean parameter θ. However, in many applications Σ = Σ(θ) is a function of θ and is estimated by a plug–in procedure, i.e., $\widehat{\mathbf{\Sigma }}=\mathbf{\Sigma }(\widehat{\mathit{\theta }})$. Such situations arise in nonlinear regression models, where Y_i = f(X_i; θ) + ε_i, and in generalized linear models. In all such cases, although the theoretical asymptotic variance of the $\widehat{\mathit{\theta }}$ may be non–singular matrix, its estimated variance $\mathbf{\Sigma }(\widehat{\mathit{\theta }})$ may be nearly singular. As seen in Lim et al. (2013) and in Lim (2014), the condition number (in log scale) of the estimated variance matrix can be as high as 10⁵ or more. Thus, the theory and methodology discussed in this paper are relevant to non–linear as well as generalized linear models.

Finally, as noted by a referee it is of interest to extend the current methodology within the framework of order restricted model selection (cf. Mulder et al., 2010 and Kuiper et al. 2012). An additional direction is to explore the performance of the proposed tests in high dimensions.