Sparse Estimation of Conditional Graphical Models With Application to Gene Networks

Abstract

In many applications the graph structure in a network arises from two sources: intrinsic connections and connections due to external effects. We introduce a sparse estimation procedure for graphical models that is capable of isolating the intrinsic connections by removing the external effects. Technically, this is formulated as a conditional graphical model, in which the external effects are modeled as predictors, and the graph is determined by the conditional precision matrix. We introduce two sparse estimators of this matrix using the reproduced kernel Hilbert space combined with lasso and adaptive lasso. We establish the sparsity, variable selection consistency, oracle property, and the asymptotic distributions of the proposed estimators. We also develop their convergence rate when the dimension of the conditional precision matrix goes to infinity. The methods are compared with sparse estimators for unconditional graphical models, and with the constrained maximum likelihood estimate that assumes a known graph structure. The methods are applied to a genetic data set to construct a gene network conditioning on single-nucleotide polymorphisms.

1. INTRODUCTION

Sparse estimation of the Gaussian graphical models has undergone intense development during the recent years, partly due to their wide applications in such fields as information retrieval and genomics, and partly due to the increasing maturity of statistical theories and techniques surrounding sparse estimation. See, for example, Meinshausen and Buhlmann (2006), Yuan and Lin (2007), Bickel and Levina (2008), Peng et al. (2009), Guo et al. (2009), and Lam and Fan (2009). The precursor of this line of work is the Gaussian graphical model in which the graph structure is assumed to be known; see Dempster (1972) and Lauritzen (1996).

Let Y = (Y¹, …, Y^p)^⊤ be a random vector, Γ = {1, …, p}, and E ⊆ Γ × Γ. Let = (Γ, E) be the graph with its vertices in Γ and edges in E. We say that Y follows a Gaussian graphical model (GGM) with respect to (Lauritzen 1996) if Y has a multivariate normal distribution and

$${Y^i}_{⫫}{Y}^j\mid Y^{-(i,j)},(i,j)\in E^c,$$ (1)

where A ⫫ B|C means A and B are independent given C, and Y^−(i,j) denotes the set {Y¹, …, Y^p}\{Yⁱ, Y^j}. Let ω_ij be the (i, j)th entry of [var(Y)]⁻¹. Then, Yⁱ ⫫ Y^j|Y^−(i,j) if and only if ω_ij = 0. Thus, estimating E is equivalent to estimating the set {(i, j) ∈ Γ × Γ: ω_ij = 0}. Conventional Gaussian graphical models focus on maximum likelihood estimation of ω_ij given the knowledge of E. In sparse estimation of graphical models, however, E is itself estimated by sparse regularization, such as lasso and adaptive lasso (Tibshirani 1996; Zou 2006).

The conditional graphical model, with which we are concerned, is motivated by the analysis of gene networks and the regulating effects of DNA markers. Let X¹, …, X^q represent the genetic markers at q locations in a genome, and let Y¹, …, Y^p represent the expression levels of p genes. The objective is to infer how the q genetic markers affect the expression levels of the p genes and how these p genes affect each other. Since some markers may have regulating effects on more than one gene, the connections among the genes are of two kinds: the connections due to shared regulation by the same marker, and the innate connections among the genes aside from their shared regulators. In this setting, we are interested in identifying the network of genes after removing the effects from shared regulations by the markers.

The situation is illustrated by Figure 1, in which X represents a single marker, and Y¹, Y², Y³ represent the expressions of three genes. If we consider the marginal distribution of the random vector (Y¹, Y², Y³), then there are two (undirected) edges in the unconditional graphical model: 1 ↔ 2 and 2 ↔ 3, as represented by the solid and the dotted line segments. However, if we condition on the marker and consider the conditional distribution of Y¹, Y², Y³|X, then there is only one (undirected) edge, 2 ↔ 3, in the conditional graphical model, as represented by the solid line segment.

Figure 1.

Unconditional and conditional graphical models.

In mathematical terms, a conditional graphical model can be represented by

$${Y^i}_{⫫}{Y}^j\mid \{Y^{-(i,j)},\mathbf{X}\}\text{for}(i,j)\notin E,$$ (2)

where X = (X¹, …, X^q)^⊤. Compared with the unconditional graphical model, here we have an additional random vector X, whose effects we would like to remove when constructing the network for Y¹, …, Y^p. The conditional graphical model (2) was introduced by Lafferty, McCallum, and Pereira (2001) under the name “conditional random field.” However, in that article, the graph = (Γ, E) was assumed known and maximum likelihood was used to estimate the relevant parameters.

Our strategy for estimating the conditional graphical model (2) is as follows. In the first step, we propose a class of flexible and easy-to-implement initial (nonsparse) estimators for the conditional variance matrix Σ = var(Y | X), based on a conditional variance operator between the reproducing kernel Hilbert spaces (RKHS) of X and Y. In the second step, we incorporate the nonsparse estimators with two types of sparse penalties, the lasso and the adaptive lasso, to obtain sparse estimators of the conditional precision matrix Σ⁻¹.

We have chosen RKHS as our method to estimate Σ for several reasons. First, it does not require a rigid regression model for Y versus X. Second, the dimension of X only appears through a kernel function, so that the dimension of the largest matrix we need to invert is the sample size n, regardless of the dimension of X. This feature is particularly attractive when we deal with a large number of predictors. Finally, RKHS provides a natural mechanism to impose regularization on regression. Here, it is important to realize that our problem involves two kinds of regularization: one for the components of var(Y | X) and the other for the regression of Y on X. Considering the nature of our problem, the former must be sparse, but the latter need not be. Indeed, since estimating the regression parameter is not our purpose, it seems more natural to introduce the regularization for regression through RKHS than to try to parameterize the regression and then regularize the parameters.

The rest of the article is organized as follows. In Section 2, we introduce a conditional variance operator in RKHS and describe its relation with the conditional gaussian graphical model. In Section 3, we derive two RKHS-based estimators of conditional variance Σ. In Section 4, we subject the RKHS estimators to sparse penalties to estimate the conditional precision matrix Θ = Σ⁻¹. In Sections 5, 6, and 7, we establish the asymptotic properties of the sparse estimators for a fixed dimension p. In Section 8, we derive the convergence rate of the sparse estimators when p goes to infinity with the sample size. In Section 9, we discuss some issues involved in implementation. In Sections 10 and 11, we investigate and explore the performance of the proposed methods through simulation and data analysis.

2. CONDITIONAL VARIANCE OPERATOR IN RKHS

We begin with a formal definition of the conditional Gaussian graphical model. Throughout this article, we use to denote expectation to avoid confusion with the edge set E. We use ℙ to denote probability measures.

Definition 1

We say that (X, Y) follows a conditional Gaussian graphical model (CGGM) with respect to a graph = (Γ, E) if

1. relation (2) holds;

2. Y | X ~ N( (Y | X), Σ) for some nonrandom, positive-definite matrix Σ.

Note that we do not assume a regression model for Y versus X. However, we do require that Y | X is multivariate normal with a constant conditional variance, which is satisfied if Y = f (X) + ε, ε ⫫ X, and ε ~ N(0, Σ) for an arbitrary f.

Let Ω_X ⊆ ℝ^q and Ω_Y ⊆ ℝ^p be the support of X and Y. Let κ_X: Ω_X × Ω_X → ℝ, κ_Y: Ω_Y × Ω_Y → ℝ be positive-definite kernels and and be their corresponding RKHS’s. For further information about RKHS and the choices of kernels, see Aronszajn (1950) and Vapnik (1998). For two Hilbert spaces and , and a bounded bilinear form b: × → ℝ, there uniquely exist bounded linear operators A: → and B: → such that 〈f, Bg〉 = 〈Af, g〉 = b(f, g) for any f ∈ and g ∈ . Applying this fact to the bounded bilinear forms

$$cov[f_1(\mathbf{X}),f_2(\mathbf{X})],cov[g_1(\mathbf{Y}),g_2(\mathbf{Y})],cov[f(\mathbf{X}),g(\mathbf{Y})],$$

we obtain three bounded linear operators

$$\begin{array}{l}{\mathrm{\sum }}_{XX}:{\mathcal{H}}_X\to {\mathcal{H}}_X,{\mathrm{\sum }}_{YY}:{\mathcal{H}}_Y\to {\mathcal{H}}_Y,\\ {\mathrm{\sum }}_{XY}:{\mathcal{H}}_Y\to {\mathcal{H}}_X.\end{array}$$ (3)

Furthermore, Σ_XY can be factorized as ${\mathrm{\sum }}_{XX}^{\frac{1}{2}}{R}_{XY}{\mathrm{\sum }}_{YY}^{\frac{1}{2}}$, where R_XY: → is a uniquely defined bounded linear operator. The conditional variance operator of Y, given X, is then defined as the bounded operator from to :

$${\mathrm{\sum }}_{YY\mid X}={\mathrm{\sum }}_{YY}-{\mathrm{\sum }}_{YY}^{\frac{1}{2}}{R}_{YX}{R}_{XY}{\mathrm{\sum }}_{YY}^{\frac{1}{2}}.$$

This construction is due to Fukumizu, Bach, and Jordan (2009).

Let L₂(ℙ_X) denote the class of functions of X that are squared integrable with respect to ℙ_X, + ℝ denote the set of functions {h + c: h ∈ , c ∈ ℝ}, and cl(·) denote the closure of a set in L₂(ℙ_X). The next theorem describes how the conditional operator Σ_{YY | X} uniquely determines the CGGM, and suggests a way to estimate Σ.

Theorem 1

Suppose

1. ⊆ L₂(ℙ_X) and, for each i = 1, …, p, (Yⁱ|X) ∈ cl( + ℝ);

2. κ_Y is the linear kernel: κ_Y (a, b) = 1 + a^⊤b, where a, b ∈ ℝ^p.

Then,

$$\mathbf{\sum }=\{{\langle y^r,{\mathrm{\sum }}_{YY\mid X}{y}^s\rangle }_{{\mathcal{H}}_Y}:r,s=1,\dots ,p\}.$$ (4)

The assumption ⊆ L₂(ℙ_X) is satisfied if the function κ_X(x, x) belongs to L₂(ℙ_X), which is a mild requirement.

Proof

By assumption 2, any member of can be written as α^⊤y for some α ∈ ℝ^p. Hence, by Proposition 2 of Fukumizu et al. (2009),

$$\begin{array}{l}{\langle {\mathit{\alpha }}^{\top }\mathbf{y},{\mathrm{\sum }}_{YY\mid X}({\mathit{\alpha }}^{\top }\mathbf{y})\rangle }_{{\mathcal{H}}_Y}=\underset{f\in {\mathcal{H}}_X+ℝ}{inf}var[{\mathit{\alpha }}^{\top }\mathbf{Y}-f(\mathbf{X})]\\ =\mathbb{E}[var({\mathit{\alpha }}^{\top }\mathbf{Y}\mid \mathbf{X})]+\underset{f\in {\mathcal{H}}_X+ℝ}{inf}var\{\mathbb{E}[({\mathit{\alpha }}^{\top }\mathbf{Y}\mid \mathbf{X})-f(\mathbf{X})]\}.\end{array}$$

By assumption 1, for any ε > 0, there is an f ∈ + ℝ such that var{ [(α^⊤Y | X) − f (X)]} < ε. So, the second term on the right-hand side above is 0, and we have

$${\langle {\mathit{\alpha }}^{\top }y,{\mathrm{\sum }}_{YY\mid X}({\mathit{\alpha }}^{\top }y)\rangle }_{{\mathcal{H}}_Y}=\mathbb{E}[var({\mathit{\alpha }}^{\top }\mathbf{Y}\mid \mathbf{X})].$$ (5)

In the meantime, we note that

$$\begin{array}{l}{\langle y^i+y^j,{\mathrm{\sum }}_{YY\mid X}(y^i+y^j)\rangle }_{{\mathcal{H}}_Y}-{\langle y^i-y^j,{\mathrm{\sum }}_{YY\mid X}(y^i-y^j)\rangle }_{{\mathcal{H}}_Y}\\ =4{\langle y^i,{\mathrm{\sum }}_{YY\mid X}{y}^j\rangle }_{{\mathcal{H}}_Y},\mathbb{E}[var(Y^i+Y^j\mid \mathbf{X})]\\ -\mathbb{E}[var(Y^i-Y^j\mid \mathbf{X})]=4\mathbb{E}[cov(Y^i,Y^j\mid \mathbf{X})].\end{array}$$

Applying Equation (5) to the left-hand sides of the above equations, we obtain

$${\langle y^i,{\mathrm{\sum }}_{YY\mid X}{y}^j\rangle }_{{\mathcal{H}}_Y}=\mathbb{E}[cov(Y^i,Y^j\mid \mathbf{X})]=cov(Y^i,Y^j\mid \mathbf{X}),$$

as desired.

Condition 1 in the theorem can be replaced by the stronger assumption that + ℝ is a dense subset of L₂(P_X), which is satisfied by some well known kernels, such as the Gaussian radial kernel. See Fukumizu et al. (2009).

3. TWO RKHS ESTIMATORS OF CONDITIONAL COVARIANCE

To construct a sample estimate of Equation (4), we need to represent operators as matrices in a finite-dimensional space. To this end, we first introduce a coordinate notation system, adopted from Horn and Johnson (1985, p. 31) with slight modifications. Let be a finite-dimensional Hilbert space and = {b₁, …, b_m}⊆ be a set of functions that span but may not be linearly independent. We refer to as a spanning system (as opposed to a basis). Any f ∈ can be written as α₁b₁ + ··· + α_mb_m for some α = (α₁, …, α_m)^⊤ ∈ ℝ^m. This vector is denoted by [f], and is called a -coordinate of f. Note that [f] is not unique unless b₁, …, b_m are linearly independent. However, this does not matter because ${\sum }_{i=1}^m{({[f]}_{\mathcal{B}})}_i{b}_i$ unique regardless of the form of [f]. The same reasoning also applies to the nonuniqueness of coordinates below.

Let and be finite-dimensional Hilbert spaces with spanning systems = {b₁₁, …, b_1m1}and = {b₂₁, …, b_2m2}. Let T : → be a linear operator. The ( , )-representation of T, denoted by [T], is the m₂ × m₁ matrix

$$\{{({[T{\mathrm{b}}_{1i}]}_{{\mathcal{B}}_2})}_j:j=1,\dots ,m_2,i=1,\dots ,m_1\}.$$

Then, ( [T] )[f] is a -coordinate of Tf. We write this relation as

$${[Tf]}_{{\mathcal{B}}_2}=({{}_{{\mathcal{B}}_2}[T]}_{{\mathcal{B}}_1}){[f]}_{{\mathcal{B}}_1}.$$ (6)

Let be another finite-dimensional Hilbert space with a spanning system . Let T₁: → and T₂ : → be linear operators. Then,

$${{}_{{\mathcal{B}}_3}[T_2{T}_1]}_{{\mathcal{B}}_1}=({{}_{{\mathcal{B}}_3}[T_2]}_{{\mathcal{B}}_2})({{}_{{\mathcal{B}}_2}[T_1]}_{{\mathcal{B}}_1}).$$ (7)

The equality means the right-hand side is a ( , )-representation of T₂T₁. Finally, if is a finite-dimensional Hilbert space with a spanning system and T : → is a self-adjoint and positive-semidefinite linear operator, then, for any c > 0,

$$\mathcal{B}{[T^c]}_{\mathcal{B}}={({{}_{\mathcal{B}}[T]}_{\mathcal{B}})}^c.$$ (8)

Let (X₁, Y₁), …, (X_n, Y_n) be independent copies of (X, Y), ℙ_n be the empirical measure based on this sample, and be the integral with respect to ℙ_n. Let

$$\begin{array}{l}{\widehat{\mathcal{H}}}_X=\text{span}\{{\kappa }_X(·,{\mathbf{X}}_i)-{\mathbb{E}}_n{\kappa }_X(·,\mathbf{X}):i=1,\dots ,n\},\\ {\widehat{\mathcal{H}}}_Y=\text{span}\{{\kappa }_Y(·,{\mathbf{Y}}_i)-{\mathbb{E}}_n{\kappa }_Y(·,\mathbf{Y}):i=1,\dots ,n\},\end{array}$$ (9)

where, for example, κ_X(·, X) stands for the function x ↦ κ_X(x, X). We center the functions κ_X(·, X_i) because constants do not play a role in our development. Let Σ̂_XX, Σ̂_YY, and Σ̂_XY be as defined in the last section but with , replaced by , , and cov replaced by the sample covariance. Let and denote the spanning systems in Equation (9). Let K_X and K_Y represent the n × n kernel matrices {κ_X(X_i, X_j)} and {κ_Y (Y_i, Y_j)}. For a symmetric matrix, A, let A^† represent its Moore-Penrose inverse. Let Q_n = I_n − n ⁻¹J_n, I_n is the n × n identity matrix, and J_n be the n × n matrix whose entries are 1. To simplify notation, we abbreviate Q_n by Q throughout the rest of the article. The following lemma crystallizes some known results, which can be proved by coordinate manipulation via formulas (6), (7), and (8).

Lemma 1

The following relations hold:

1. [Σ̂ _XX] = n⁻¹QK_XQ, [Σ̂_YY]= n⁻¹QK_YQ;

2. [Σ̂_YX]= n⁻¹QK_XQ, [Σ̂_XY]= n⁻¹QK_Y Q;

3. [Σ̂_{YY | X}] = n⁻¹[QK_Y Q − (QK_XQ)(QK_XQ)^† (QK_Y Q)].

When the dimension q of X is large relative to n, it is beneficial to use regularized version of (QK_XQ)^†. Here, we employ two types of regularization: the principal-component (PC) regularization and the ridge-regression (RR) regularization. Let A be a positive-semidefinite matrix with eigenvalues λ₁, …, λ _n and eigen-vectors v₁, …, v_n. Let ε ≥0. We call the matrix ${(\mathbf{A})}_{\epsilon }^{†}={\sum }_{i=1}^n{\lambda }_i^{-1}{\mathbf{v}}_i{\mathbf{v}}_i^{\top }I({\lambda }_i>\epsilon )$ the PC-inverse of A. Note that ${(\mathbf{A})}_0^{†}={\mathbf{A}}^{†}$. We call the matrix ${(\mathbf{A})}_{\epsilon }^{‡}={(\mathbf{A}+\epsilon {\mathbf{I}}_n)}^{-1}$ the RR-inverse of A. The following result can be verified by simple calculation.

Lemma 2

For any f₁, f₂ ∈ , we have 〈 f₁, f₂〉 = ([f₁])^⊤QK_Y Q[f₂].

We now derive the sample estimate of the conditional covariance matrix Σ= var(Y | X). Let D_Y denote the matrix (Y₁, …, Y_n)^⊤.

Theorem 2

Let Σ̂_{YY | X}: → be defined by the coordinate representation

$${{}_{{\mathcal{B}}_Y}[{\widehat{\mathrm{\sum }}}_{YY\mid X}]}_{{\mathcal{B}}_Y}=n^{-1}\left[{\mathbf{QK}}_Y\mathbf{Q}-\left({\mathbf{QK}}_X\mathbf{Q}\right){({\mathbf{QK}}_X\mathbf{Q})}_{{\epsilon }_n}^{\ast }({\mathbf{QK}}_Y\mathbf{Q})\right],$$ (10)

where * can be either † or ‡, K_Y is the linear kernel matrix, and {ε_n} is a sequence of nonnegative numbers. Then

$$\{{\langle y^i,{\widehat{\mathrm{\sum }}}_{YY\mid X}{y}^j\rangle }_{{\widehat{\mathcal{H}}}_Y}\}=n^{-1}[{\mathbf{D}}_Y^{\top }{\mathbf{QD}}_Y-{\mathbf{D}}_Y^T\mathbf{Q}\left({\mathbf{QK}}_X\mathbf{Q}\right){({\mathbf{QK}}_X\mathbf{Q})}_{{\epsilon }_n}^{\ast }{\mathbf{QD}}_Y].$$ (11)

Despite its appearance, the matrix $({\mathbf{QK}}_X\mathbf{Q}){({\mathbf{QK}}_X\mathbf{Q})}_{\epsilon }^{\ast }$ is actually a symmetric matrix. One can also show that Equation (11) is a positive-semidefinite matrix. We denote the matrix (11) as Σ̂_PC(ε_n) if * = †, and as Σ̂_RR(ε_n) if * = ‡. In the following, e_i represents a p-dimensional vector whose ith entry is 1 and other entries are 0. For an expression that represents a vector, such as [f], let ([f])_i denote its ith entry.

Proof of Theorem 1

Note that, for an f ∈ , [f] is any vector a ∈ ℝ^p such that f (y) = a^⊤QD_Y y. Because $y^i={\mathbf{e}}_i^{\top }\mathbf{y}={\mathbf{e}}_i^{\top }{({\mathbf{D}}_Y^{\top }{\mathbf{QD}}_Y)}^{-1}{\mathbf{D}}_Y^{\top }{\mathbf{QD}}_Y\mathbf{y}$, we have

$${[y^i]}_{{\mathcal{B}}_Y}={\mathbf{D}}_Y{\left({\mathbf{D}}_Y^{\top }{\mathbf{QD}}_Y\right)}^{-1}{\mathbf{e}}_i.$$

Then, by Lemma 2,

$${\langle y^i,{\widehat{\mathrm{\sum }}}_{YY\mid X}{y}^j\rangle }_{{\mathcal{H}}_Y}={\mathbf{e}}_i^{\top }{\left({\mathbf{D}}_Y^{\top }{\mathbf{QD}}_Y\right)}^{-1}{\mathbf{D}}_Y^{\top }{\mathbf{QK}}_Y\mathbf{Q}({{}_{{\mathcal{B}}_Y}[{\widehat{\mathrm{\sum }}}_{YY\mid X}]}_{{\mathcal{B}}_Y}){\mathbf{D}}_Y{\left({\mathbf{D}}_Y^{\top }{\mathbf{QD}}_Y\right)}^{-1}{\mathbf{e}}_j.$$

When κ_Y is the linear kernel, ${\mathbf{K}}_Y={\mathbf{D}}_Y{\mathbf{D}}_Y^{\top }$. Hence, the right-hand side reduces to

$${\mathbf{e}}_i^{\top }{\mathbf{D}}_Y^{\top }\mathbf{Q}({{}_{{\mathcal{B}}_Y}[{\widehat{\mathrm{\sum }}}_{YY\mid X}]}_{{\mathcal{B}}_Y}){\mathbf{D}}_Y{\left({\mathbf{D}}_Y^{\top }{\mathbf{QD}}_Y\right)}^{-1}{\mathbf{e}}_j.$$

Now substitute Equation (10) into the above expression to complete the proof.

4. INTRODUCING SPARSE PENALTY

Let Σ̂ denote either of the RKHS estimators given by Equation (11). We are interested in the sparse estimation of Θ= Σ⁻¹. Let

$$L_n(\mathbf{\Theta })=-logdet(\mathbf{\Theta })+\text{tr}(\mathbf{\Theta }\widehat{\mathbf{\sum }}).$$ (12)

This objective function has the same form as that used in unconditional Gaussian graphical models, such as those considered by Lauritzen (1996), Yuan and Lin (2007), and Friedman, Hastie, and Tibshirani (2008), except that the sample covariance matrix therein is replaced by the RKHS estimate of conditional covariance matrix.

To achieve sparsity in Θ, we introduce two types of penalized versions of the objective function (12):

$$\text{lasso}:{\mathrm{\Upsilon }}_n(\mathbf{\Theta })=L_n(\mathbf{\Theta })+{\lambda }_n\sum _{i\ne j}\mid {\theta }_{ij}\mid ,$$ (13)

$$\text{adaptive}\text{lasso}:{\mathrm{\Lambda }}_n(\mathbf{\Theta })=L_n(\mathbf{\Theta })+{\lambda }_n\sum _{i\ne j}{\mid {\stackrel{\sim }{\theta }}_{ij}\mid }^{-\gamma }\mid {\theta }_{ij}\mid ,$$ (14)

where, in Equation (14), γ is a positive number and {θ̃_ij} is a $\sqrt{n}$-consistent, nonsparse estimate of Θ. For more details about the development of these two types of penalty functions; see Tibshirani (1996), Zou (2006), Zhao and Yu (2006), and Zou and Li (2008). Yuan and Lin (2007) used lasso and nonnegative garrote (Breiman 1995) penalty functions for the unconditional graphical model, the latter of which is similar to adaptive lasso with γ = 1. In the following, we will write

$$P_n(\mathbf{\Theta })={\lambda }_n\sum _{i\ne j}\mid {\theta }_{ij}\mid ,{\mathrm{\Pi }}_n(\mathbf{\Theta })={\lambda }_n\sum _{i\ne j}{\mid {\stackrel{\sim }{\theta }}_{ij}\mid }^{-\gamma }\mid {\theta }_{ij}\mid .$$

5. VON MISES EXPANSIONS OF THE RKHS ESTIMATORS

The sparse and oracle properties of the estimators introduced in Section 4 depend heavily on the asymptotic properties of the RKHS estimators Σ̂_PC(ε_n) and Σ̂_RR(ε_n). In this section, we derive their von Mises expansions (von Mises 1947). For simplicity, we base our asymptotic development on the polynomial kernel. That is, we let

$${\kappa }_X(\mathbf{a},\mathbf{b})={({\mathbf{a}}^{\top }\mathbf{b}+1)}^r,r=1,2,\dots$$ (15)

For a matrix A and an integer k ≥ 2, let A^⊗k be the k-fold Kronecker product A ⊗ ··· ⊗ A. For k = 1, we adopt the convention A^⊗1 = A. For a k-dimensional array B = {b_i1…ik : i₁, …, i_k = 1, …, q}, we define the “vec half” operator as

$$\text{vech}(\mathbf{B})=\{b_{i_1\cdots i_k}:1\le i_k\le \cdots \le i_1\le q\}.$$

This is a generalization of the vech operator for matrices introduced by Henderson and Searle (1979). It can be shown that there exists a matrix G_k,q, of full column rank, such that vec(B) = G_k,q vech(B). The specific form of G_k,q is not important to us. For k = 1, we adopt the convention vech(B) = B, G_1,q = I_q. Let

$$\begin{array}{l}\mathbf{U}=\left(\begin{array}{c}\text{vech}({\mathbf{X}}^{\otimes 1})\\ ⋮\\ \text{vech}({\mathbf{X}}^{\otimes r})\end{array}\right),\\ {\mathbf{U}}_i=\left(\begin{array}{c}\text{vech}({\mathbf{X}}_i^{\otimes 1})\\ ⋮\\ \text{vech}({\mathbf{X}}_i^{\otimes r})\end{array}\right),\text{and}{\mathbf{D}}_U={({\mathbf{U}}_1,\dots ,{\mathbf{U}}_n)}^{\top }.\end{array}$$

The estimators Σ̂_PC(ε_n) and Σ̂_RR(ε_n) involve n × n matrices QK_XQ and ${({\mathbf{QK}}_X\mathbf{Q})}_{{\epsilon }_n}^{\ast }$, which are difficult to handle asymptotically because their dimensions grow with n. We now give an asymptotically equivalent expression which only involves matrices of fixed dimensions. Let

$$\mathbf{\sum }^{\sim }=n^{-1}\left[{\mathbf{D}}_Y^{\top }{\mathbf{QD}}_Y-{\mathbf{D}}_Y^{\top }{\mathbf{QD}}_U{\left({\mathbf{D}}_U^{\top }{\mathbf{QD}}_U\right)}^{-1}{\mathbf{D}}_U^{\top }{\mathbf{QD}}_Y\right].$$ (16)

Lemma 3

Suppose κ_X is the polynomial kernel (15) and var(U) is positive definite.

1. If ε_n = o(n), then, with probability tending to 1, Σ̂_PC(ε_n) = Σ̃;

2. If ${\epsilon }_n=o(n^{\frac{1}{2}})$, then ${\widehat{\mathbf{\sum }}}_{\text{RR}}({\epsilon }_n)=\mathbf{\sum }^{\sim }+o_P(n^{-\frac{1}{2}})$.

It is interesting to note that different choices of inversion requires different convergence rates for ε_n.

Proof

By simple computation, we find

$${\left({\mathbf{X}}_i^{\top }{\mathbf{X}}_j+1\right)}^r=1+\sum _{k=1}^r\left(\begin{array}{c}r\\ k\end{array}\right){\left[\text{vech}\left({\mathbf{X}}_i^{\otimes k}\right)\right]}^{\top }{\mathbf{G}}_{k,q}^{\top }{\mathbf{G}}_{k,q}\text{vech}\left({\mathbf{X}}_i^{\otimes k}\right).$$

Hence, ${\mathbf{K}}_X={\mathbf{J}}_n+{\mathbf{D}}_U\mathbf{C}{\mathbf{D}}_U^{\top }$ where $\mathbf{C}=\text{diag}(\left(\begin{array}{c}r\\ 1\end{array}\right){\mathbf{G}}_{1,q}^{\top }{\mathbf{G}}_{1,q},\dots ,\left(\begin{array}{c}r\\ r\end{array}\right){\mathbf{G}}_{r,q}^{\top }{\mathbf{G}}_{r,q})$. So,

$${\mathbf{QK}}_X\mathbf{Q}={\mathbf{QD}}_U{\mathbf{CD}}_U^{\top }\mathbf{Q}.$$ (17)

Let s be the dimension of U. Then, rank(QK_XQ) = s. Let λ₁ ≥ ··· ≥ λ_s > 0 be the nonzero eigenvalues and v₁, …, v_s be the corresponding vectors of this matrix. Since λ₁,…, λ_s are also eigenvalues of $n(n^{-1}{\mathbf{C}}^{\frac{1}{2}}{\mathbf{D}}_U^{\top }{\mathbf{QD}}_U{\mathbf{C}}^{\frac{1}{2}})$, which converges in probability to the positive-definite matrix ${\mathbf{C}}^{\frac{1}{2}}var(\mathbf{U}){\mathbf{C}}^{\frac{1}{2}}$, we have λ_s = nc + o_P(n) for some c > 0. This, together with ε_n = o(n), implies ${({\mathbf{QK}}_X\mathbf{Q})}_{{\epsilon }_n}^{†}={({\mathbf{QD}}_U{\mathbf{CD}}_U^{\top }\mathbf{Q})}^{†}$ with probability tending to 1. Consequently, with probability tending to 1,

$${\widehat{\mathbf{\sum }}}_{\text{PC}}({\epsilon }_n)=n^{-1}\left[{\mathbf{D}}_Y^{\top }{\mathbf{QD}}_Y-{\mathbf{D}}_Y^{\top }\mathbf{Q}\left({\mathbf{QD}}_U{\mathbf{CD}}_U^{\top }\mathbf{Q}\right){\left({\mathbf{QD}}_U{\mathbf{CD}}_U^{\top }\mathbf{Q}\right)}^{†}{\mathbf{QD}}_Y\right].$$ (18)

Now it is easy to verify that if B ∈ ℝ^s×t is a matrix of full column rank and A ∈ ℝ^t×t is a positive-definite matrix, then

$$({\mathbf{BAB}}^{\top }){({\mathbf{BAB}}^{\top })}^{†}=\mathbf{B}{({\mathbf{B}}^{\top }\mathbf{B})}^{-1}{\mathbf{B}}^{\top }.$$ (19)

Thus, the right-hand side of Equation (18) is Σ̃, which proves part 1. To prove part 2, we first note that

$$({\mathbf{QK}}_X\mathbf{Q}){({\mathbf{QK}}_X\mathbf{Q})}_{{\epsilon }_n}^{‡}=\left({\mathbf{QK}}_X\mathbf{Q}-\sum _{i=1}^s\frac{{\lambda }_i{\epsilon }_n}{{\lambda }_i+{\epsilon }_n}{\mathbf{v}}_i{\mathbf{v}}_i^{\top }\right){({\mathbf{QK}}_X\mathbf{Q})}^{†}.$$ (20)

Since the function −ε_n/(ε_n + λ) is increasing for λ> 0, we have

$$-\frac{{\epsilon }_n}{{\lambda }_s+{\epsilon }_n}({\mathbf{QK}}_X\mathbf{Q}){({\mathbf{QK}}_X\mathbf{Q})}^{†}\le ({\mathbf{QK}}_X\mathbf{Q}){({\mathbf{QK}}_X\mathbf{Q})}_{{\epsilon }_n}^{‡}-({\mathbf{QK}}_X\mathbf{Q}){({\mathbf{QK}}_X\mathbf{Q})}^{†}\le 0.$$

Hence,

$$\begin{array}{l}0\ge n^{-1}{\mathbf{D}}_Y^{\top }\mathbf{Q}\left[({\mathbf{QK}}_X\mathbf{Q}){({\mathbf{QK}}_X\mathbf{Q})}_{{\epsilon }_n}^{‡}-({\mathbf{QK}}_X\mathbf{Q}){({\mathbf{QK}}_X\mathbf{Q})}^{†}\right]{\mathbf{QD}}_Y\\ \ge -\frac{{\epsilon }_n}{{\lambda }_s+{\epsilon }_n}{n}^{-1}{\mathbf{D}}_Y^{\top }\mathbf{Q}({\mathbf{QK}}_X\mathbf{Q}){({\mathbf{QK}}_X\mathbf{Q})}^{†}{\mathbf{QD}}_Y,\end{array}$$ (21)

By Equation (19),

$$n^{-1}{\mathbf{D}}_Y^{\top }\mathbf{Q}({\mathbf{QK}}_X\mathbf{Q}){({\mathbf{QK}}_X\mathbf{Q})}^{†}{\mathbf{QD}}_Y=\left(n^{-1}{\mathbf{D}}_Y^{\top }{\mathbf{QD}}_U\right){\left(n^{-1}{\mathbf{D}}_U^{\top }{\mathbf{QD}}_U\right)}^{-1}(n^{-1}{\mathbf{D}}_U{\mathbf{QD}}_Y)\stackrel{P}{\to }cov(\mathbf{Y},\mathbf{U}){[var(\mathbf{U})]}^{-1}cov(\mathbf{U},\mathbf{Y}).$$

Hence, the right-hand side of Equation (21) is of the order $o_P(n^{-\frac{1}{2}})$. In other words,

$${\widehat{\mathbf{\sum }}}_{\text{RR}}({\epsilon }_n)=n^{-1}\left[{\mathbf{D}}_Y^{\top }{\mathbf{QD}}_Y-{\mathbf{D}}_Y^{\top }\mathbf{Q}({\mathbf{QD}}_U{\mathbf{CD}}_U^{\top }\mathbf{Q}){({\mathbf{QD}}_U{\mathbf{CD}}_U^{\top }\mathbf{Q})}^{†}{\mathbf{QD}}_Y\right]+o_P\left(n^{-\frac{1}{2}}\right).$$

Here, we evoke Equation (19) again to complete the proof.

Note that when r = 1 and p < n, and ε_n = 0, Σ̂_PC(ε_n) reduces to

$$n^{-1}\left[{\mathbf{D}}_Y^{\top }{\mathbf{QD}}_Y-{\mathbf{D}}_Y{\mathbf{QD}}_X{\left({\mathbf{D}}_X^{\top }{\mathbf{QD}}_X\right)}^{-1}{\mathbf{D}}_X{\mathbf{QD}}_Y\right]={var}_n(\mathbf{Y})-{cov}_n(\mathbf{Y},\mathbf{X}){[{var}_n(\mathbf{X})]}^{-1}{cov}_n(\mathbf{X},\mathbf{Y}),$$

where var_n(·) and cov_n(·, ·) denote the sample variance and covariance matrices. This is exactly the sample estimate of the residual variance for linear regression.

Lemma 3 allows us to derive the asymptotic expansions of Σ̂_PC(ε_n) and Σ̂_RR(ε_n) from that of Σ̃, which is a (matrix-valued) function of sample moments. Let be a convex family of probability measures defined on Ω_XY that contains all the empirical distributions ℙ_n and the true distribution ℙ₀ of (X, Y). Let T : → ℝ^p×p be the following statistical functional:

$$\mathbb{P}\mapsto {var}_{\mathbb{P}}(\mathbf{Y})-{cov}_{\mathbb{P}}(\mathbf{Y},\mathbf{U}){[{var}_{\mathbb{P}}(\mathbf{U})]}^{-1}{cov}_{\mathbb{P}}(\mathbf{U},\mathbf{Y}).$$ (22)

In this notation, Σ̃ = T (ℙ_n). In the following, we use var and cov to denote the variance and covariance under ℙ₀. For the polynomial kernel (15), the evaluation T (ℙ₀) has a special meaning, as described in the next lemma. Its proof is standard, and is omitted.

Lemma 4

Suppose:

1. Entries of (Y | X) are polynomials in X¹, …, X^q of degrees no more than r;

2. Y | X ~ N( (Y | X), Σ), where Σ is a nonrandom matrix.

Then, T (ℙP₀) = Σ.

Let ℙ_α = (1 − α)ℙ₀ + αℙ_n, where α ∈ [0, 1]. Let D_α denote the differential operator ∂/∂α, and let D_α=0 denote the operation of taking derivative with respect to α and then evaluating the derivative at α = 0. It is well known that if T is Hadamard differentiable with respect to the norm || · ||_∞ in , then

$$T({\mathbb{P}}_n)=T({\mathbb{P}}_0)+D_{\alpha =0}T({\mathbb{P}}_{\alpha })+o_P\left(n^{-\frac{1}{2}}\right).$$ (23)

Since the functional T in our case is a smooth function of sample moments, it is Hadamard differentiable under very general conditions. See, for example, Reeds (1976), Fernholz (1983), Bickel et al. (1993), and Ren and Sen (1991). The next lemma can be verified by straightforward computation.

Lemma 5

Let V₁ and V₂ be square-integrable random vectors. Then,

$$D_{\alpha =0}[{cov}_{{\mathbb{P}}_{\alpha }}({\mathbf{V}}_1,{\mathbf{V}}_2)]={\mathbb{E}}_n\mathbf{q}({\mathbf{V}}_1,{\mathbf{V}}_2)-{\mathbb{E}}_{\mathbf{q}}({\mathbf{V}}_1,{\mathbf{V}}_2),$$

where q(V₁, V₂) = (V₁ − V₁)(V₂ − V₂)^⊤.

We will adopt the following notational system:

$$\begin{array}{l}{\mathit{\mu }}_Y=\mathbb{E}(\mathbf{Y}),{\mathit{\mu }}_U=\mathbb{E}(\mathbf{U}),{\mathbf{V}}_Y=var(\mathbf{Y}),{\mathbf{V}}_U=var(\mathbf{U}),\\ {\mathbf{V}}_{UY}=cov(\mathbf{U},\mathbf{Y}).\end{array}$$

The next theorem gives the first-order von-Mises expansion of Σ̃.

Theorem 3

Suppose the functional in Equation (22) is Hadamard differentiable, and the conditions in Lemma 4 hold. Then,

$$\mathbf{\sum }^{\sim }=\mathbf{\sum }+{\mathbb{E}}_n\mathbf{M}+o_P\left(n^{-\frac{1}{2}}\right),$$ (24)

where M = M₀ − M₀ and

$${\mathbf{M}}_0=\left[\mathbf{Y}-{\mathit{\mu }}_Y-{\mathbf{V}}_{YU}{\mathbf{V}}_U^{-1}(\mathbf{U}-{\mathit{\mu }}_U)\right]\times {\left[\mathbf{Y}-{\mathit{\mu }}_Y-{\mathbf{V}}_{YU}{\mathbf{V}}_U^{-1}(\mathbf{U}-{\mathit{\mu }}_U)\right]}^{\top }.$$

Proof

We have

$$D_{\alpha }T({\mathbb{P}}_{\alpha })=D_{\alpha }[{var}_{{\mathbb{P}}_{\alpha }}(\mathbf{Y})]-D_{\alpha }\{{cov}_{{\mathbb{P}}_{\alpha }}(\mathbf{Y},\mathbf{U})\times {[{var}_{{\mathbb{P}}_{\alpha }}(\mathbf{U})]}^{-1}{cov}_{{\mathbb{P}}_{\alpha }}(\mathbf{U},\mathbf{Y})\}.$$

We now apply Lemma 5 to obtain

$$\begin{array}{l}{D}_{\alpha =0}\{{cov}_{{\mathbb{P}}_{\alpha }}(\mathbf{Y},\mathbf{U}){[{var}_{{\mathbb{P}}_{\alpha }}(\mathbf{U})]}^{-1}{cov}_{{\mathbb{P}}_{\alpha }}(\mathbf{U},\mathbf{Y})\}\\ =[{\mathbb{E}}_n\mathbf{q}(\mathbf{Y},\mathbf{U})-\mathbb{E}\mathbf{q}(\mathbf{Y},\mathbf{U})]{\mathbf{V}}_U^{-1}{\mathbf{V}}_{UY}+{\mathbf{V}}_{YU}{D}_{\alpha =0}\\ \times \{{[{var}_{{\mathbb{P}}_{\alpha }}(\mathbf{U})]}^{-1}\}{\mathbf{V}}_{UY}+{\mathbf{V}}_{YU}{\mathbf{V}}_U^{-1}[{\mathbb{E}}_n\mathbf{q}(\mathbf{U},\mathbf{Y})-\mathbb{E}\mathbf{q}(\mathbf{U},\mathbf{Y})].\end{array}$$

By the chain rule for differentiation and Lemma 5,

$$D_{\alpha =0}\{{[{var}_{{\mathbb{P}}_{\alpha }}(\mathbf{U})]}^{-1}\}=-{\mathbf{V}}_U^{-1}[{\mathbb{E}}_n\mathbf{q}(\mathbf{U},\mathbf{U})-\mathbb{E}\mathbf{q}(\mathbf{U},\mathbf{U})]{\mathbf{V}}_U^{-1}.$$

Hence,

$$\begin{array}{l}{D}_{\alpha =0}T({\mathbb{P}}_{\alpha })=[{\mathbb{E}}_n\mathbf{q}(\mathbf{Y},\mathbf{Y})-\mathbb{E}\mathbf{q}(\mathbf{Y},\mathbf{Y})]\\ -[{\mathbb{E}}_n\mathbf{q}{(\mathbf{Y},\mathbf{U})}^{\top }-\mathbb{E}\mathbf{q}(\mathbf{Y},\mathbf{U})]{\mathbf{V}}_U^{-1}{\mathbf{V}}_{UY}\\ +{\mathbf{V}}_{YU}{\mathbf{V}}_U^{-1}[{\mathbb{E}}_n\mathbf{q}(\mathbf{U},\mathbf{U})-\mathbb{E}\mathbf{q}(\mathbf{U},\mathbf{U})]{\mathbf{V}}_U^{-1}{\mathbf{V}}_{UY}\\ -{\mathbf{V}}_{YU}{\mathbf{V}}_U^{-1}[{\mathbb{E}}_n\mathbf{q}(\mathbf{U},\mathbf{Y})-\mathbb{E}\mathbf{q}(\mathbf{U},\mathbf{Y})].\end{array}$$

This can be rewritten as M₀ − M₀, where M₀ is the matrix in the theorem.

Using this expansion, we can write down the asymptotic distribution of Σ̃.

Corollary 1

Suppose the functional in Equation (22) is Hadamard differentiable. Then,

$$\sqrt{n}\text{vec}(\mathbf{\sum }^{\sim }-\mathbf{\sum })\stackrel{\mathcal{D}}{\to }N(0,\mathbf{\Xi }),$$ (25)

where

$$\mathbf{\Xi }={\left({\mathbf{I}}_p,-{\mathbf{V}}_{YU}{\mathbf{V}}_U^{-1}\right)}^{\otimes 2}var\left[\left(\begin{array}{c}\mathbf{Y}-{\mathit{\mu }}_Y\\ \mathbf{U}-{\mathit{\mu }}_U\end{array}\right)\otimes \left(\begin{array}{c}\mathbf{Y}-{\mathit{\mu }}_Y\\ \mathbf{U}-{\mathit{\mu }}_U\end{array}\right)\right]{\left(\begin{array}{c}{\mathbf{I}}_p\\ -{\mathbf{V}}_U^{-1}{\mathbf{V}}_{UY}\end{array}\right)}^{\otimes 2}.$$

By Lemma 3, Theorem 3, and Slutsky’s theorem, we arrive at the following von Mises expansions for the two RKHS estimators Σ̂_PC(ε_n) and Σ̂_RR(ε_n) of the conditional variance Σ.

Corollary 2

Suppose the functional in Equation (22) is Hadamard differentiable, and the conditions in Lemma 4 hold.

1. If ${\epsilon }_n=o(n^{\frac{1}{2}})$, then Σ̂_RR(ε_n) has expansion (24).

2. If ε_n = o(n), then Σ̂_PC(ε_n) has expansion (24).

Although in this article we have only studied the asymptotic distribution for the polynomial kernel, the basic formulation and analysis could potentially be extended to other kernels under some regularity conditions. We leave this to future research.

6. SPARSITY AND ASYMPTOTIC DISTRIBUTION: THE LASSO

In this section, we study the asymptotic properties of the sparse estimator based on the objective function ϒ_n(Θ) in Equation (13). Let denote the class of all p × p symmetric matrices. For a matrix A ∈ , let σ_i (A) be the ith eigenvalue of A. Note that, for any integer r, we have

$$\text{tr}({\mathbf{A}}^r)=\sum _{i=1}^p{\sigma }_i^r(\mathbf{A}).$$ (26)

Let E_n be the sample estimate of E; that is, E_n = {(i, j): θ̂_ij ≠ 0}, where Θ̂ = {θ̂_ij}is the minimizer of Equation (13). Following Lauritzen (1996), for a matrix A = {a_ij} and a graph = (Γ, E), let A( ) denote the matrix that sets a_ij to 0 whenever (i, j) ∉ E. Let

$$\begin{array}{l}{ℝ}^{p\times p}(\mathcal{G})=\{\mathbf{A}(\mathcal{G}):\mathbf{A}\in {ℝ}^{p\times p}\},{\mathbb{S}}^{p\times p}(\mathcal{G})\\ =\{\mathbf{A}(\mathcal{G}):\mathbf{A}\in {\mathbb{S}}^{p\times p}\}.\end{array}$$

We define a bivariate sign function as follows. For any numbers a, b, let

$$\text{sign}(a,b)=\text{sign}(a)I(a\ne 0)+\text{sign}(b)I(a=0).$$

The following lemma will prove useful. Its proof is omitted.

Lemma 6

The bivariate sign function sign(a, b) has the following properties:

1. For any c₁ > 0, c₂ > 0, sign(c₁a, c₂b) = sign(a, b);

2. For sufficiently small |b|, |a + b| − |a| = sign(a, b)b.

The next theorem generalizes Theorem 1 of Yuan and Lin (2007). It applies to any random matrix Σ̂ with expansion (24), including Σ̂_PC(ε_n) and Σ̂_RR(ε_n).

Theorem 4

Suppose (X, Y) follows CGGM with respect to a graph (Γ, E). Let Θ̂ be the minimizer of Equation (13), where $\sqrt{n}{\lambda }_n\to {\lambda }_0>0$ and Σ̂ having the expansion (24). Let W ∈ ℝ^p×p be a random matrix such that vec(W) is distributed as N[0, var(vec(M))]. Then, the following assertions hold.

1. 0 < lim_{n →∞} ℙ(E_n = E) < 1;

2. For any ε > 0, there is a λ₀ > 0 such that lim_{n →∞} ℙ(E_n = E) > 1 − ε;

3. $\sqrt{n}(\widehat{\mathbf{\Theta }}-{\mathbf{\Theta }}_0)\stackrel{\mathcal{D}}{\to }\text{argmin}\{\mathrm{\Phi }(\mathbf{\Delta },\mathbf{W}):\mathbf{\Delta }\in {\mathbb{S}}^{p\times p}\}$, where

   $$\mathrm{\Phi }(\mathbf{\Delta },\mathbf{W})=\text{tr}(\mathbf{\Delta }\mathbf{\sum }\mathbf{\Delta }\mathbf{\sum }/2+\mathbf{\Delta }\mathbf{W})+{\lambda }_0\sum _{i\ne j}\text{sign}({\theta }_{0,ij},{\delta }_{ij}){\delta }_{ij}.$$

The inequality lim_n→∞ ℙ(E_n = E) > 0 implies that, as n → ∞, there is a positive probability to estimate a parameter as 0 when it is 0. A stronger property is that this probability tends to 1. For clarity, we refer to the former property as sparsity, and the latter as sparsistency (see Fan and Li 2001; Lam and Fan 2009). According to this definition, the two inequalities in part 1 mean that lasso is sparse but not sparsistent. Note that the unconstrained maximum likelihood estimate—and indeed any regular estimate—is not sparse. Part 2 means that, even though lasso is not sparsistent, we can make it as close to sparsistent as we wish by choosing a sufficiently large λ₀. Part 3 gives the asymptotic distribution of $\sqrt{n}(\widehat{\mathbf{\Theta }}-{\mathbf{\Theta }}_0)$. It is not the same as that of the maximum likelihood estimate under the constraint θ_ij = 0 for (i, j) ∈ E^c. That is, it does not have the oracle property. The proof of part 3 is similar to that of Theorem 1 in Yuan and Lin (2007) in the context of GGM. However, to our knowledge there were no previous results parallel to parts 1 and 2 for GGM. Knight and Fu (2000) contains some basic ideas for the asymptotics for lassotype estimators.

Proof of Theorem 2

We prove the three assertions in the order 3, 1, 2.

3. Let Θ₀ be the true value of Θ, and let

$${\mathrm{\Phi }}_n(\mathbf{\Delta })=n[{\mathrm{\Upsilon }}_n({\mathbf{\Theta }}_0+n^{-1/2}\mathbf{\Delta })-{\mathrm{\Upsilon }}_n({\mathbf{\Theta }}_0)].$$

By an argument similar to Yuan and Lin (2007, Theorem 1), it can be shown that

$$\begin{array}{l}n[L_n({\mathbf{\Theta }}_0+n^{-1/2}\mathbf{\Delta })-L_n({\mathbf{\Theta }}_0)]=\text{tr}(\mathbf{\Delta }\mathbf{\sum }\mathbf{\Delta }\mathbf{\sum })/2+n^{1/2}\text{tr}(\mathbf{\Delta }{\mathbb{E}}_n\mathbf{M}),\\ n[P_n({\mathbf{\Theta }}_0+n^{-1/2}\mathbf{\Delta })-P_n({\mathbf{\Theta }}_0)]={\lambda }_0\sum _{i\ne j}\text{sign}({\theta }_{0,ij},{\delta }_{ij}){\delta }_{ij}+o(1).\end{array}$$

Since $n^{1/2}{\mathbb{E}}_n\mathbf{M}\stackrel{\mathcal{D}}{\to }\mathbf{W}$, we have

$${\mathrm{\Phi }}_n(\mathbf{\Delta })\stackrel{\mathcal{D}}{\to }\text{tr}(\mathbf{\Delta }\mathbf{\sum }\mathbf{\Delta }\mathbf{\sum }/2+\mathbf{\Delta }\mathbf{W})+{\lambda }_0\sum _{i\ne j}\text{sign}({\theta }_{0,ij},{\delta }_{ij}){\delta }_{ij}=\mathrm{\Phi }(\mathbf{\Delta },\mathbf{W}).$$

Both Φ_n(Δ) and Φ(Δ, W) are strictly convex with probability 1. Applying Theorem 4.4 of Geyer (1994) we see that

$$\text{argmin}\{{\mathrm{\Phi }}_n(\mathbf{\Delta }):\mathbf{\Delta }\in {\mathbb{S}}^{p\times p}\}\stackrel{\mathcal{D}}{\to }\text{argmin}\{\mathrm{\Phi }(\mathbf{\Delta },\mathbf{W}):\mathbf{\Delta }\in {\mathbb{S}}^{p\times p}\}.$$

However, by construction, if Δ̂ is the (almost surely unique) minimizer of Φ_n(Δ), then Δ̂ = n^1/2(Θ̂ − Θ₀). This proves part 3.

1. For a generic function f(t) defined on t ∈ ℝ^s, let ${\partial }_{t_i}^L$ and ${\partial }_{t_i}^R$ be the left and right partial derivatives with respect to the ith component of t. When f is differentiable with respect to t_i, we write ${\partial }_{t_i}={\partial }_{t_i}^L={\partial }_{t_i}^R$. Note that E_n = E if and only if Φ(Δ, W) is minimized within ( ). This happens if and only if

$${\partial }_{{\delta }_{ij}}^L\mathrm{\Phi }(\mathbf{\Delta },\mathbf{W})\le 0\le {\partial }_{{\delta }_{ij}}^R\mathrm{\Phi }(\mathbf{\Delta },\mathbf{W}),(i,j)\in \mathrm{\Gamma }\times \mathrm{\Gamma },i\ge j,\text{and}\mathbf{\Delta }\in {\mathbb{S}}^{p\times p}(\mathcal{G}).$$ (27)

Here, we only consider the cases i ≥ j because Δ is a symmetric matrix. Let

$$\begin{array}{l}L(\mathbf{\Delta },\mathbf{W})=\text{tr}(\mathbf{\Delta }\mathbf{\sum }\mathbf{\Delta }\mathbf{\sum }/2+\mathbf{\Delta }\mathbf{W}),\\ P(\mathbf{\Delta },\mathbf{W})={\lambda }_0\sum _{i\ne j}\text{sign}({\theta }_{0,ij},{\delta }_{ij}){\delta }_{ij}.\end{array}$$

Then, ∂L(Δ, W)/∂Δ = ΣΔΣ + W. For (i, j) ∈ E, P (Δ, W) is differentiable with respect to δ_ij and ∂_δij P(Δ, W) = λ₀sign(θ_0,ij). For (i, j) ∈ E^c, P (Δ, W) is not differentiable with respect to δ_ij, but has left and right derivatives, given by ${\partial }_{{\delta }_{ij}}^{L}P(\mathbf{\Delta },\mathbf{W})=-{\lambda }_0$ and ${\partial }_{{\delta }_{ij}}^{R}P(\mathbf{\Delta },\mathbf{W})={\lambda }_0$. Condition (27) now reduces to

$$\{\begin{array}{ll}{(\mathbf{\sum }\mathbf{\Delta }\mathbf{\sum })}_{ij}+w_{ij}+s_{ij}=0,\hfill & \text{if}(i,j)\in E,i\ge j,\hfill \\ {(\mathbf{\sum }\mathbf{\Delta }\mathbf{\sum })}_{ij}+w_{ij}\in [-{\lambda }_0,{\lambda }_0],\hfill & \text{if}(i,j)\in E^c,i\ge j,\hfill \end{array}$$ (28)

where Δ ∈ ( ), s_ij = λ₀sign(θ_0,ij) if (i, j) ∈ E and i ≠ j and s_ij = 0 if i = j.

Now consider the event

$$G=\{\{w_{ij}:i\ge j\}:\text{Equation}(28)\text{is}\text{satisfied}\text{for}\text{some}\mathbf{\Delta }\in {\mathbb{S}}^{p\times p}(\mathcal{G})\}.$$

We need to show that ℙ(G) > 1. Since Δ belongs to ( ), it has as many free parameters as there are equations in the first line of Equation (28). Since Σ is a nonsingular matrix, for any {w_ij: i ≥ j, (i, j) ∈ E}, there exists a unique Δ that satisfies the first line of Equation (28) and it is a linear function of {w_ij: (i, j) ∈ E, i ≥ j}. Writing this function as Δ({w_ij : (i, j) ∈ E, i ≥ j}), we see that Equation (28) is satisfied if and only if

$${[\mathbf{\sum }\mathbf{\Delta }(\{w_{\mu \nu }:(\mu ,\nu )\in E,\mu \ge \nu \})\mathbf{\sum }]}_{ij}+w_{ij}\in [-{\lambda }_0,{\lambda }_0],\text{for}i\ge j,(i,j)\in E^c.$$

Since the mapping

$$\tau :\{w_{ij}:i\ge j\}\mapsto \{2{[\mathbf{\sum }\mathbf{\Delta }(\{w_{\mu \nu }:(\mu ,\nu )\in E,\mu \ge \nu \})\mathbf{\sum }]}_{ij}+2w_{ij}:i\ge j,(i,j)\in E^c\}$$

from ℝ^p(p+1) to ℝ^card(Ec)/2 is continuous, the set τ⁻¹[(−λ₀, λ₀)^card(Ec)/2] is open in ℝ^p(p+1)/2. Furthermore, this open set is nonempty because if we let

$$w_{ij}=-{[\mathbf{\sum }\mathbf{\Delta }(\{w_{\mu \nu }:(\mu ,\nu )\in E,\mu \ge \nu \})\mathbf{\sum }]}_{ij},\text{for}(i,j)\in E^c,i\ge j,$$

then τ ({w_ij : i ≥ j}) = 0. Because {w_ij : i ≥ j} has a multivariate normal distribution with a nonsingular covariance matrix, any nonempty open set in ℝ^p(p+1)/2 has positive probability. This proves lim_n→∞ ℙ(E_n = E) > 0.

Similarly, the set τ⁻¹[(λ₀, 3λ₀)^card(Ec)/2] is open in ℝ^p(p+1)/2, and if we let

$$w_{ij}=-{[\mathbf{\sum }\mathbf{\Delta }(\{w_{\mu \nu }:(\mu ,\nu )\in E,\mu \ge \nu \})\mathbf{\sum }]}_{ij}+2{\lambda }_0,\text{for}(i,j)\in E^c,i\ge j,$$

then τ({w_ij: i ≥ j}) = (2λ₀, …, 2λ₀)^⊤ ∈ (λ₀, 3λ₀)^card(Ec)/2. Hence, the W-probability of τ⁻¹[(λ₀, 3λ₀)^card(Ec)/2] is positive, implying lim_n→∞ ℙ(E_n = E) < 1.

2. For each (i, j) ∈ E^c, i ≥ j, Let U_ij = [ΣΔ ({w_μν : (μ, ν) ∈ E, μ ≥ ν}) Σ]_ij + w_ij. Then, for any η > 0, there is a ${\lambda }_0^{ij}>0$ such that $\mathbb{P}(U_{ij}\in [-{\lambda }_0^{ij},{\lambda }_0^{ij}])>1-\eta$. Let ${\lambda }_0=max\{{\lambda }_0^{ij}:(i,j)\in E^c,i\ge j\}$, i ≥ j}. Then, ℙ(U_ij ∈ [−λ₀, λ₀]) > 1 − η. Hence,

$$\begin{array}{l}\underset{n\to \infty }{lim}\mathbb{P}(E_n=E)=\mathbb{P}(U_{ij}\in [-{\lambda }_0,{\lambda }_0],(i,j)\in E^c,i\ge j)\\ \ge 1-\sum _{(i,j)\in E^c,i\ge j}\{1-\mathbb{P}(U_{ij}\in [-{\lambda }_0,{\lambda }_0])\}\\ \ge 1-\text{card}(E^c)\eta /2.\end{array}$$

This proves part 2 because η can be arbitrarily small.

7. SPARSISTENCY AND ORACLE PROPERTY: THE ADAPTIVE LASSO

We now turn to the adaptive lasso based on Equation (14). We still use Θ̂ = {θ̂_ij} to denote the minimizer of Equation (14). For a matrix Δ = {δ_ij} ∈ , and a set C ∈ Γ × Γ, let δ_C = {δ_ij : (i, j) ∈ C, i ≥ j}, which is to be interpreted as a vector where index i moves first, followed by index j.

Theorem 5

Suppose that (X, Y) follows CGGM with respect to a graph (Γ, E). Let Θ̂ be the minimizer of Equation (14), where Σ̂ has expansion (24), and

$$\underset{n\to \infty }{lim}{n}^{1/2}{\lambda }_n=0,\underset{n\to \infty }{lim}{n}^{(1+\gamma )/2}{\lambda }_n=\infty .$$

Suppose Θ̃ in Equation (14) is a $\sqrt{n}$-consistent estimate of Θ₀ with ℙ(θ̃_ij ≠ 0) = 1 for (i, j) ∈ E^c. Then,

1. lim_n→∞ ℙ(E_n = E) = 1;

2. $\sqrt{n}(\widehat{\mathbf{\Theta }}-{\mathbf{\Theta }}_0)\stackrel{\mathcal{D}}{\to }\text{argmin}\{L(\mathbf{\Delta },\mathbf{G});\mathbf{\Delta }\in {\mathbb{S}}^{p\times p}(\mathcal{G})\}$.

Part 1 asserts that the adaptive lasso is sparsistent; part 2 asserts that it is asymptotically equivalent to the minimizer of L(Δ, G) when E is known. In this sense Θ̂ is oracle. The condition P (θ̃_ij ≠ 0) = 1 means that θ̃_ij is not sparse, which guarantees that |θ̃_ij |^−γ is well defined. For example, we can use the inverse of one of the RKHS estimates as Θ̃.

Proof

Let Δ ∈ and Ψ_n(Δ) = n[Λ_n(Θ₀ + n^−1/2Δ) − Λ_n(Θ₀)]. Consider the difference

$$n[{\mathrm{\Pi }}_n({\mathbf{\Theta }}_0+n^{-1/2}\mathbf{\Delta })-{\mathrm{\Pi }}_n({\mathrm{\Theta }}_{\mathbf{0}})]={\lambda }_{n}n\sum _{i\ne j}{\mid {\stackrel{\sim }{\theta }}_{ij}\mid }^{-\gamma }(\mid {\theta }_{0,ij}+n^{-1/2}{\delta }_{ij}\mid -\mid {\theta }_{0,ij}\mid ).$$

By Lemma 6, for large enough n, the right-hand side can be rewritten as

$$\sum _{i\ne j}{n}^{1/2}{\lambda }_n{\mid {\stackrel{\sim }{\theta }}_{ij}\mid }^{-\gamma }\text{sign}({\theta }_{0,ij},n^{-1/2}{\delta }_{ij}){\delta }_{ij}.$$ (29)

If (i, j) ∈ E, then |θ̃_ij |^−γ = O_P(1). Because n^1/2 λ_n → 0, the summand for such (i, j) converges to 0 in probability. Hence, Equation (29) reduces to

$$\sum _{(i,j)\in E^c}{n}^{1/2}{\lambda }_n{\mid {\stackrel{\sim }{\theta }}_{ij}\mid }^{-\gamma }\text{sign}({\delta }_{ij}){\delta }_{ij}+o_P(1)=\sum _{(i,j)\in E^c}{n}^{1/2}{\lambda }_n{\mid {\stackrel{\sim }{\theta }}_{ij}\mid }^{-\gamma }\mid {\delta }_{ij}\mid +o_P(1).$$ (30)

If (i, j) ∉ E, then θ̃_ij = O_P(n^−1/2), and hence

$$n^{1/2}{\lambda }_n{\mid {\stackrel{\sim }{\theta }}_{ij}\mid }^{-\gamma }={\lambda }_n{n}^{(1-\gamma )/2}{\mid n^{1/2}{\stackrel{\sim }{\theta }}_{ij}\mid }^{-\gamma }=\frac{{\lambda }_n{n}^{(1+\gamma )/2}}{O_P(1)}\stackrel{P}{\to }\infty .$$

From this, we see that Equation (30) converges in probability to ∞ unless δ_Ec = 0, in which case it converges in probability to 0. In other words,

$$n[{\mathrm{\Pi }}_n({\mathbf{\Theta }}_0+n^{-1/2}\mathbf{\Delta })-{\mathrm{\Pi }}_n({\mathbf{\Theta }}_0)]\stackrel{P}{\to }\{\begin{array}{ll}0,\hfill & {\mathit{\delta }}_{E^c}=0,\hfill \\ \infty ,\hfill & {\mathit{\delta }}_{E^c}\ne 0,\hfill \end{array}$$

This implies that

$${\mathrm{\Psi }}_n(\mathbf{\Delta })\stackrel{\mathcal{D}}{\to }\mathrm{\Psi }(\mathbf{\Delta },\mathbf{G}),\text{where}\mathrm{\Psi }(\mathbf{\Delta },\mathbf{G})=\{\begin{array}{ll}L(\mathbf{\Delta },\mathbf{G}),\hfill & {\mathit{\delta }}_{E^c}=0,\hfill \\ \infty ,\hfill & {\mathit{\delta }}_{E^c}\ne 0.\hfill \end{array}$$

Since both Ψ_n(Δ) and Ψ(Δ, G) are convex and Ψ(Δ, G) has a unique minimum, by the epi-convergence results of Geyer (1994), we have

$$\text{argmin}\{{\mathrm{\Psi }}_n(\mathbf{\Delta }):\mathbf{\Delta }\in {\mathbb{S}}^{p\times p}\}\stackrel{\mathcal{D}}{\to }\text{argmin}\{\mathrm{\Psi }(\mathbf{\Delta },\mathbf{G}):\mathbf{\Delta }\in {\mathbb{S}}^{p\times p}\}.$$

This proves part 2 because the right-hand side is, in fact, argmin{L(Δ, G) : Δ ∈ ( )}, and the left-hand side is $\sqrt{n}(\widehat{\mathbf{\Theta }}-{\mathbf{\Theta }}_0)$.

The function Ψ(Δ, W) is always minimized in a region of Δ in which it is not ∞. As a consequence, if Δ minimizes Ψ(Δ, W) over , then δ_Ec = 0. Hence, ℙ(E_n ⊇ E) → 1. In the meantime, since $\widehat{\mathbf{\Theta }}-{\mathbf{\Theta }}_0=O_P(n^{-\frac{1}{2}})$, we have ${\widehat{\theta }}_{ij}={\theta }_{0,ij}+O_P(n^{-\frac{1}{2}})$. If (i, j) ∈ E, then θ_0,ij ≠ 0. Thus, we see that ℙ(E_n ⊆ E) → 1. This proves part 1.

We now derive the explicit expression of the asymptotic distribution of Θ̂. Let G be the unique matrix in ℝ^{p2×[card(E)+p]/2} such that for any Δ ∈ ( ), vec(Δ) = Gδ_E. For example, if p = 3 and E = {(1, 1), (2, 2), (3, 3), (1, 2), (2, 1), (1, 3), (3, 1)}, then δ_E = (δ₁₁, δ₂₁, δ₃₁, δ₂₂, δ₃₃)^⊤, and G is defined by

$${\mathbf{G}}^{\top }=\left(\begin{array}{ccccccccc}1& 0& 0& 0& 0& 0& 0& 0& 0\\ 0& 1& 0& 1& 0& 0& 0& 0& 0\\ 0& 0& 1& 0& 0& 0& 1& 0& 0\\ 0& 0& 0& 0& 1& 0& 0& 0& 0\\ 0& 0& 0& 0& 0& 0& 0& 0& 1\end{array}\right).$$

Corollary 3

Under the assumptions of Theorem 5,

$$\sqrt{n}\text{vec}(\widehat{\mathbf{\Theta }}-{\mathbf{\Theta }}_0)\stackrel{\mathcal{D}}{\to }N(0,\mathbf{V}),$$

where

$$\mathbf{V}=\mathbf{G}{[{\mathbf{G}}^{\top }({\mathbf{\sum }}^2\otimes {\mathbf{I}}_p)\mathbf{G}]}^{-1}{\mathbf{G}}^{\top }\mathbf{\Xi }\mathbf{G}{[{\mathbf{G}}^{\top }({\mathbf{\sum }}^2\otimes {\mathbf{I}}_p)\mathbf{G})]}^{-1}{\mathbf{G}}^{\top }.$$ (31)

Proof

Note that

$$\begin{array}{l}\text{tr}(\mathbf{\Delta }\mathbf{\sum }\mathbf{\Delta }\mathbf{\sum })={\text{vec}}^{\top }(\mathbf{\Delta }\mathbf{\sum })\text{vec}(\mathbf{\Delta }\mathbf{\sum })={\text{vec}}^{\top }(\mathbf{\Delta })({\mathbf{\sum }}^2\otimes {\mathbf{I}}_p)\text{vec}(\mathbf{\Delta })\\ \text{tr}(\mathbf{\Delta }\mathbf{W})={\text{vec}}^{\top }(\mathbf{W})\text{vec}(\mathbf{\Delta }).\end{array}$$

Hence,

$$L(\mathbf{\Delta },\mathbf{W})={\mathit{\delta }}_E^{\top }{\mathbf{G}}^{\top }({\mathbf{\sum }}^2\otimes {\mathbf{I}}_p)\mathbf{G}{\mathit{\delta }}_E/2+{\text{vec}}^{\top }(\mathbf{W})\mathbf{G}{\mathit{\delta }}_E.$$

This is a quadratic function minimized by δ_E = −[G^⊤(Σ² ⊗ I_p)G]⁻¹G^⊤vec(W). In terms of Δ, the minimizer is vec(Δ) = −G[G^⊤(Σ² ⊗ I_p)G]⁻¹G^⊤vec(W). The corollary now follows from Theorem 5.

The asymptotic variance of $\sqrt{n}(\widehat{\mathbf{\Theta }}-{\mathbf{\Theta }}_0)$ can be estimated by replacing the moments in Equation (25) and (31) by their sample estimates.

8. CONVERGENCE RATE FOR HIGH-DIMENSIONAL GRAPH

In the last two sections, we have studied the asymptotic properties of our sparse CGGM estimators with the number of nodes p in the graph held fixed. We now investigate the case where p = p_n tends to infinity. Due to the limited space, we shall focus on the lasso-penalized RKHS estimator with PC regularization. That is, the minimizer of ϒ_n(Θ) with Σ̂ in Equation (12) is taken to be the RKHS estimator Σ̂_PC(ε_n) based on κ_X(a, b) = (1 + a^⊤b)^r. Throughout this section, Θ̂ denotes this estimator. The large-p_n convergence rate for the lasso estimator of the (unconditional) GGM has been studied by Rothman et al. (2008).

In this case, Σ, Θ, and Y should in principle be written as Σ⁽ⁿ⁾, Θ⁽ⁿ⁾, and Y⁽ⁿ⁾ because they now depend on n. However, to avoid complicated notation, we still use Σ, Θ, and Y, keeping in mind their dependence on n. Following Rothman et al. (2008), we develop the convergence rate in Frobenius norm. Let ||·||₁, ||·||_F, and ||·||_∞ be the L₁-norm, Frobenius norm, and the L_∞-norm of a matrix A ∈ ℝ^d₁×d₂:

$$\begin{array}{l}{\Vert \mathbf{A}\Vert }_1=\sum _{i=1}^{d_1}\sum _{j=1}^{d_2}\mid a_{ij}\mid ,{\Vert \mathbf{A}\Vert }_F={\left(\sum _{i=1}^{d_1}\sum _{j=1}^{d_2}{a}_{ij}^2\right)}^{1/2},\\ {\Vert \mathbf{A}\Vert }_{\infty }=max\mid a_{ij}\mid ,\end{array}$$

where a_ij denotes the (i, j)th entry of A. Let ρ(A) denote the number of nonzero entries of A. For easy reference, we list some properties of these matrix functions in the following proposition.

Proposition 1

Let A ∈ ℝ^d₁×d₂, B ∈ ℝ^d₂×d₃. Then,

$$\begin{array}{l}{\Vert \mathbf{AB}\Vert }_{\infty }\le d_2{\Vert \mathbf{A}\Vert }_{\infty }{\Vert \mathbf{B}\Vert }_{\infty },\mid \text{tr}(\mathbf{AB})\mid \le {\Vert \mathbf{A}\Vert }_1{\Vert \mathbf{B}\Vert }_{\infty },\\ {\Vert \mathbf{A}\Vert }_1\le \sqrt{\rho (\mathbf{A})}{\Vert \mathbf{A}\Vert }_F.\end{array}$$

The first inequality follows from the definition of ||·||_∞; the second from Hölder’s inequality; the third from the Cauchy-Schwarz inequality. The last two inequalities were used in Rothman et al. (2008). Let ℕ = {1, 2, …}. Consider the array of random matrices: { ${\mathbf{A}}_i^{(n)}:i=1,\dots ,n$, n ∈ ℕ}, where A⁽ⁿ⁾ ∈ ℝ^p_n×q, p_n may depend on n but q is fixed. Let (A⁽ⁿ⁾)_rs denote the (r, s)th entry of A⁽ⁿ⁾.

Lemma 7

Let { ${\mathbf{A}}_i^{(n)}:i=1,\dots ,n$, n ∈ ℕ} be an array of random matrices in ℝ^p_n×q, each of whose rows is an iid sample of a random matrix A⁽ⁿ⁾. Suppose that the moment generating functions of (A⁽ⁿ⁾)_st, say φ_nst, are finite on an interval (−δ, δ), and their second derivatives are uniformly bounded over this interval for all s = 1,…, p_n, t = 1,…, q, n ∈ ℕ. If p_n → ∞, then ${\Vert {\mathbb{E}}_n({\mathbf{A}}^{(n)})-\mathbb{E}({\mathbf{A}}^{(n)})\Vert }_{\infty }+O_P(log{p}_n/\sqrt{n})$.

Proof

Let μ_nst = (A⁽ⁿ⁾)_st. Since $\mid {\mu }_{\mathit{nst}}\mid \le 1+{\phi }_{\mathit{nst}}^{″}(0)$, there is C > 0 such that |μ_nst| ≤ C for all s, t, n. Let (B⁽ⁿ⁾)_st = (A⁽ⁿ⁾)_st − μ_nst, and ψ_nst be the moment generating function of (B⁽ⁿ⁾)_st. Then, ψ_nst (τ) = e^−μ_nstτφ_nst (τ). For any a > 0,

$$\begin{array}{l}\mathbb{P}\left(\sqrt{n}\left({\mathbb{E}}_n{\left({\mathbf{B}}^{(n)}\right)}_{st}\right)>a\right)=\mathbb{P}\left(e^{\sqrt{n}{\mathbb{E}}_n{({\mathbf{B}}^{(n)})}_{st}}>e^a\right)\\ \le e^{-a}{\left[{\psi }_{\mathit{nst}}\left(n^{-1/2}\right)\right]}^n.\end{array}$$

By Taylor’s theorem and noticing that ${\psi }_{\mathit{nst}}^{\prime }(0)=0$, we have

$${\psi }_{\mathit{nst}}(n^{-1/2})=1+{\psi }_{\mathit{nst}}^{″}({\xi }_{\mathit{nst}})/(2n)\le 1+e^C{\phi }_{\mathit{nst}}^{″}({\xi }_{\mathit{nst}})/(2n)$$

for some 0 ≤ ξ_nst ≤ n^−1/2. By assumption, there is C₁ > 0 such that lim sup_n→∞ φ″ (ξ_nst) ≤ C₁ for all s = 1,…, p_n and t = 1,…, q. Hence,

$${[{\psi }_{\mathit{nst}}(n^{-1/2})]}^n\le {[1+e^C{C}_1/(2n)]}^n\to e^{e^C{C}_1/2}\equiv C_2.$$

Thus, we have $lim{sup}_{n\to \infty }P(\sqrt{n}{\mathbb{E}}_n{({\mathbf{B}}^{(n)})}_{st}>a)\le e^{-a}{C}_2$. By the same argument, we can show that $lim{sup}_{n\to \infty }P(\sqrt{n}{\mathbb{E}}_n{({\mathbf{B}}^{(n)})}_{st}<-a)\le e^{-a}{C}_2$. Therefore,

$$\underset{n\to \infty }{limsup}P\left(\sqrt{n}\mid {\mathbb{E}}_n{\left({\mathbf{B}}^{(n)}\right)}_{st}\mid >a\right)\le 2e^{-a}{C}_2.$$

It follows that

$$\begin{array}{l}\underset{n\to \infty }{limsup}\mathbb{P}\left({\Vert {\mathbb{E}}_n({\mathbf{B}}^{(n)})\Vert }_{\infty }>c_n\right)\le \underset{n\to \infty }{limsup}\sum _{s=1}^{p_n}\sum _{t=1}^q\mathbb{P}\left(\sqrt{n}\mid {\mathbb{E}}_n[{\left({\mathbf{A}}^{(n)}\right)}_{st}\mid >\sqrt{n}{c}_n\right)\\ \le 2{qC}_2{e}^{-\sqrt{n}{c}_n+log{p}_n}.\end{array}$$

In particular,

$$\underset{n\to \infty }{limsup}\mathbb{P}\left({\Vert {\mathbb{E}}_n\left({\mathbf{B}}^{(n)}\right)\Vert }_{\infty }>2log{p}_n/\sqrt{n}\right)\le {qC}_2/p_n\to 0,$$

which implies the desired result.

In the following, we call any array of random matrices satisfying the conditions in Lemma 7 a standard array. We now establish the convergence rate of ||Θ̂ −Θ₀||_F. Let s_n denote the number of nonzero off-diagonal entries of Θ₀. Let Z = Y − μ_Y − E(Y | X). Let X^t and Y^s denote the components of X and Y. Their powers are denoted by (X^t)^r and (Y^s)^r. For a symmetric matrix A, let σ_max(A) and σ_min(A) denote the maximum and minimum eigenvalues of A.

Theorem 6

Let Θ̂ be the sparse estimator defined in the first paragraph of this section with λ_n ~ (log p_n/n)^1/2, ε_n = o(n), and κ(a, b) = (1 + a^⊤b)^r. Suppose that (X, Y) follows a CGGM, and satisfies the following additional assumptions:

1. Y, YU^⊤, and ZU^⊤ are standard arrays of random matrices;

2. for all n ∈ ℕ, σ_max(Σ) < ∞ and σ_min(Σ) > 0;

3. p_n → ∞ and p_n (p_n + s_n)^1/2(log p_n)^5/2 = o(n^3/2);

4. the fixed-dimensional matrix V_U = var(U) is nonsingular;

5. each component of (Y | X = x) is a polynomial in x¹,…, x^q of at most rth order.

Then, ||Θ̂ − Θ₀||_F = O_P ([(p_n + s_n) log p_n/n]^1/2).

Note that we can allow p_n(p_n + s_n)^1/2 to get arbitrarily close to n^3/2. This condition is slightly stronger than the corresponding condition in Rothman et al. (2008) for the unconditional case, which requires (p_n + s_n) log p_n = o(n^1/2). Also note that ||Θ̂ − Θ₀||_F is the sum, instead of average, of $p_n^2$ elements (roughly p_n + s_n nonzero elements). With this in mind, the convergence rate [(p_n + s_n) log p_n/n]^1/2 is quite fast. This is the same rate as that given in Rothman et al. (2008) for the unconditional GGM.

Proof of Theorem 3

Let r_n = [(p_n + s_n) log p_n/n]^1/2, and = {Δ ∈ : ||Δ||₂ = Mr_n} for some M > 0. Let,

$${\mathbf{G}}_n(\mathbf{\Delta })={\mathrm{\Upsilon }}_n({\mathbf{\Theta }}_0+\mathbf{\Delta })-{\mathrm{\Upsilon }}_n({\mathbf{\Theta }}_0),$$

where

$${\mathrm{\Upsilon }}_n(\mathbf{\Theta })=-logdet(\mathbf{\Theta })+\text{tr}(\mathbf{\Theta }{\widehat{\mathbf{\sum }}}_{\text{PC}}({\epsilon }_n))+{\lambda }_n\sum _{i\ne j}\mid {\theta }_{ij}\mid .$$ (32)

Then, Θ̂ minimizes L(Θ) if and only if Δ̂ = Θ̂ − Θ₀ minimizes G_n(Δ). As argued by Rothman et al. (2008), since G_n(Δ) is convex in Δ and G_n(Δ̂) ≤ 0, the minimizer Δ̂ resides within the sphere if G_n(Δ) is positive and bounded away from 0 on this sphere. That is, it suffices to show

$$\mathbb{P}(inf\{G_n(\mathbf{\Delta }):\mathbf{\Delta }\in {\mathcal{D}}_n\}>0)\to 1.$$

The proof of Lemma 3 shows, in the context of fixed p, that ${({\mathbf{QK}}_X\mathbf{Q})}_{{\epsilon }_n}^{†}={({\mathbf{QD}}_U{\mathbf{CD}}_U^{\top }\mathbf{Q})}^{†}$ with probability tending to 1 if ε_n = o(n). This result still holds here because the dimension q of X remains fixed. Consequently ℙ(Σ̂_PC(ε_n) = Σ̃) → 1, where Σ̃ is as defined in Equation (16) but now its dimension increases with n. Thus, we can replace the Σ̂_PC(ε_n) in Equation (32) by Σ̃. Let

$${\widehat{\mathit{\mu }}}_U={\mathbb{E}}_n(\mathbf{U}),{\widehat{\mathbf{V}}}_{YU}=n^{-1}{\mathbf{D}}_Y^{\top }{\mathbf{QD}}_U,{\widehat{\mathbf{V}}}_U=n^{-1}{\mathbf{D}}_U^{\top }{\mathbf{QD}}_U.$$

Let ${\widehat{\mathit{\mu }}}_{Y\mid U}-{\widehat{\mathbf{V}}}_{YU}{\widehat{\mathbf{V}}}_U^{-1}(\mathbf{U}-{\widehat{\mathit{\mu }}}_U)$, and ${\mathit{\mu }}_{Y\mid U}=E(\mathbf{Y}\mid \mathbf{X})={\mathbf{V}}_{YU}{\mathbf{V}}_U^{-1}(\mathbf{U}-{\mathit{\mu }}_U)$. Then,

$$\mathbf{\sum }^{\sim }={\mathbb{E}}_n(\mathbf{Z}+{\mathit{\mu }}_{Y\mid U}-{\widehat{\mathit{\mu }}}_{Y\mid U}+{\mathit{\mu }}_Y-{\widehat{\mathit{\mu }}}_Y){(\mathbf{Z}+{\mathit{\mu }}_{Y\mid U}-{\widehat{\mathit{\mu }}}_{Y\mid U}+{\mathit{\mu }}_Y-{\widehat{\mathit{\mu }}}_Y)}^{\top }.$$

The term μ_Y|U − μ̂_Y|U can be further decomposed as Z_I + Z_II + Z_III, where

$$\begin{array}{l}{\mathbf{Z}}_I=-({\widehat{\mathbf{V}}}_{YU}-{\mathbf{V}}_{YU}){\widehat{\mathbf{V}}}_U^{-1}(\mathbf{U}-{\widehat{\mathit{\mu }}}_U),\\ {\mathbf{Z}}_{II}={\mathbf{V}}_{YU}[{\mathbf{V}}_U^{-1}(\mathbf{U}-{\mathit{\mu }}_U)-{\widehat{\mathbf{V}}}_U^{-1}(\mathbf{U}-{\widehat{\mathit{\mu }}}_U)],\\ {\mathbf{Z}}_{\mathit{III}}={\mathit{\mu }}_Y-{\widehat{\mathit{\mu }}}_Y.\end{array}$$

The function G_n(Δ) can now be rewritten as

$$G_n(\mathbf{\Delta })=L_n^{\ast }({\mathbf{\Theta }}_0+\mathbf{\Delta })-L_n^{\ast }({\mathbf{\Theta }}_0)+\sum _{v\in [I,II,\mathit{III}]}\text{tr}\left(\mathbf{\Delta }{\mathbb{E}}_n\left(\mathbf{Z}{\mathbf{Z}}_{\nu }^{\top }\right)\right)+\sum _{v\in \{I,II,\mathit{III}\}}\text{tr}(\mathbf{\Delta }{\mathbb{E}}_n({\mathbf{Z}}_v{\mathbf{Z}}^{\top }))+\sum _{v\in \{I,II,\mathit{III}\}}\sum _{\tau \in \{I,II,\mathit{III}\}}\text{tr}\left(\mathbf{\Delta }{\mathbb{E}}_n\left({\mathbf{Z}}_{\nu }{\mathbf{Z}}_{\tau }^{\top }\right)\right),$$

where $L_n^{\ast }(\mathbf{\Theta })=\text{tr}(\mathbf{\Theta }{\mathbb{E}}_n({\mathbf{ZZ}}^{\top }))-logdet(\mathbf{\Theta })+{\lambda }_n{\sum }_{i\ne j}\mid {\theta }_{ij}\mid$. Since Z ~ N(0, Σ), we can use the same argument in the proof of Theorem 1 in Rothman et al. (2008) to show that

$$\mathbb{P}\left(inf\{L_n^{\ast }({\mathbf{\Theta }}_0+\mathbf{\Delta })-L_n^{\ast }({\mathbf{\Theta }}_0):\mathbf{\Delta }\in {\mathcal{D}}_n\}>0\right)\to 1.$$

Thus, our theorem will be proved if we can show that

$$sup\{\mid \text{tr}(\mathbf{\Delta }{\mathbb{E}}_n(\mathbf{A}))\mid :\mathbf{\Delta }\in {\mathcal{D}}_n\}=o_P(1)$$

for A being any one of the following eight random matrices

$$\mathbf{Z}{\mathbf{Z}}_{\nu }^{\top },{\mathbf{Z}}_{\nu }{\mathbf{Z}}^{\top },{\mathbf{Z}}_{\nu }{\mathbf{Z}}_{\tau }^{\top },\nu ,\tau =I,II,\mathit{III}.$$ (33)

By Proposition 1, inequalities (2) and (3), we have, for Δ ∈ ,

$$\begin{array}{l}\text{tr}(\mathbf{\Delta }{\mathbb{E}}_n(\mathbf{A}))\mid \le {\Vert {\mathbb{E}}_n(\mathbf{A})\Vert }_{\infty }{\Vert \mathbf{\Delta }\Vert }_1\le {\Vert {\mathbb{E}}_n(\mathbf{A})\Vert }_{\infty }\sqrt{\rho (\mathbf{\Delta })}{\Vert \mathbf{\Delta }\Vert }_2\\ \le M{\Vert {\mathbb{E}}_n(\mathbf{A})\Vert }_{\infty }{p}_n{r}_n.\end{array}$$

Thus, it suffices to show that,

$${\Vert {\mathbb{E}}_n(\mathbf{A})\Vert }_{\infty }{p}_n{(p_n+s_n)}^{1/2}{(log{p}_n)}^{1/2}{n}^{-1/2}=o_P(1).$$ (34)

Since || (A)||_∞ = || (A^⊤)||_∞, we only need to consider the following A:

$$\mathbf{Z}{\mathbf{Z}}_I^{\top },\mathbf{Z}{\mathbf{Z}}_{II}^{\top },\mathbf{Z}{\mathbf{Z}}_{\mathit{III}}^{\top },{\mathbf{Z}}_I{\mathbf{Z}}_I^{\top },{\mathbf{Z}}_I{\mathbf{Z}}_{II}^{\top },{\mathbf{Z}}_I{\mathbf{Z}}_{\mathit{III}}^{\top },{\mathbf{Z}}_{II}{\mathbf{Z}}_{II}^{\top },{\mathbf{Z}}_{II}{\mathbf{Z}}_{\mathit{III}}^{\top },{\mathbf{Z}}_{\mathit{III}}{\mathbf{Z}}_{\mathit{III}}^{\top }.$$

From the definitions of Z and Z_I, we have

$${\mathbb{E}}_n\left(\mathbf{Z}{\mathbf{Z}}_I^{\top }\right)=-{\mathbb{E}}_n[\mathbf{Z}{(\mathbf{U}-{\mathit{\mu }}_U)}^{\top }]{\widehat{\mathbf{V}}}_U^{-1}({\widehat{\mathbf{V}}}_{UY}-{\mathbf{V}}_{UY})-{\widehat{\mathit{\mu }}}_Z{({\mathit{\mu }}_U-{\widehat{\mathit{\mu }}}_U)}^{\top }{\widehat{\mathbf{V}}}_U^{-1}({\widehat{\mathbf{V}}}_{UY}-{\mathbf{V}}_{UY}),$$ (35)

where μ̂_Z = (Z). By the first inequality of Proposition 1,

$${\Vert {\mathbb{E}}_n\left(\mathbf{Z}{\mathbf{Z}}_I^{\top }\right)\Vert }_{\infty }\le q^2{\Vert {\mathbb{E}}_n[\mathbf{Z}{(\mathbf{U}-{\mathit{\mu }}_U)}^{\top }]\Vert }_{\infty }\times {\Vert {\widehat{\mathbf{V}}}_U^{-1}\Vert }_{\infty }{\Vert {\widehat{\mathbf{V}}}_{UY}-{\mathbf{V}}_{UY}\Vert }_{\infty }+q^2{\Vert {\widehat{\mathit{\mu }}}_Z\Vert }_{\infty }{\Vert {({\mathit{\mu }}_U-{\widehat{\mathit{\mu }}}_U)}^{\top }\Vert }_{\infty }\times {\Vert {\widehat{\mathbf{V}}}_U^{-1}\Vert }_{\infty }{\Vert {\widehat{\mathbf{V}}}_{UY}-{\mathbf{V}}_{UY}\Vert }_{\infty }.$$ (36)

Since Z ⫫ X, we have E[Z(U − μ_U)^⊤] = 0. Hence, by Lemma 7,

$${\Vert {\mathbb{E}}_n[\mathbf{Z}{(\mathbf{U}-{\mathit{\mu }}_U)}^{\top }]\Vert }_{\infty }=O_P(n^{-\frac{1}{2}}log{p}_n).$$ (37)

Similarly,

$$\begin{array}{r}{\Vert {\mathbb{E}}_n[(\mathbf{Y}-{\mathit{\mu }}_Y){(\mathbf{U}-{\mathit{\mu }}_U)}^{\top }-{\mathbf{V}}_{YU}]\Vert }_{\infty }=O_P(n^{-\frac{1}{2}}log{p}_n),\\ {\Vert {\widehat{\mathit{\mu }}}_Y-{\mathit{\mu }}_Y\Vert }_{\infty }=O_P(n^{-\frac{1}{2}}log{p}_n).\end{array}$$ (38)

Since μ̂_U − μ_U has a fixed-dimension finite-variance matrix, by the central limit theorem,

$${\Vert {\widehat{\mathit{\mu }}}_U-{\mathit{\mu }}_U\Vert }_{\infty }=O_P(n^{-1/2}).$$ (39)

By definition,

$${\widehat{\mathbf{V}}}_{YU}-{\mathbf{V}}_{YU}={\mathbb{E}}_n[(\mathbf{Y}-{\mathit{\mu }}_Y){(\mathbf{U}-{\mathit{\mu }}_U)}^{\top }-{\mathbf{V}}_{YU}]-({\widehat{\mathit{\mu }}}_Y-{\mathit{\mu }}_Y){({\widehat{\mathit{\mu }}}_U-{\mathit{\mu }}_U)}^{\top }.$$

By Proposition 1, first inequality,

$${\Vert {\widehat{\mathbf{V}}}_{YU}-{\mathbf{V}}_{YU}\Vert }_{\infty }\le {\Vert {\mathbb{E}}_n[(\mathbf{Y}-{\mathit{\mu }}_Y)(\mathbf{U}-{\mathit{\mu }}_U)-{\mathbf{V}}_{YU}]\Vert }_{\infty }+{\Vert {\widehat{\mathit{\mu }}}_Y-{\mathit{\mu }}_Y\Vert }_{\infty }{\Vert {\widehat{\mathit{\mu }}}_U-{\mathit{\mu }}_U\Vert }_{\infty }.$$

Substituting Equations (38) and (39) into the above inequality, we find

$${\Vert {\widehat{\mathbf{V}}}_{YU}-{\mathbf{V}}_{YU}\Vert }_{\infty }=O_P(n^{-\frac{1}{2}}log{p}_n).$$ (40)

Since Z is multivariate normal whose components have means 0 and bounded variance, it is a standard array. Hence,

$${\Vert {\widehat{\mathit{\mu }}}_Z\Vert }_{\infty }=O_P(n^{-\frac{1}{2}}log{p}_n).$$ (41)

Since the dimension of V̂_U is fixed, its entries have finite variances, and V_U is nonsingular, we have, by the central limit theorem,

$${\Vert {\widehat{\mathbf{V}}}_U^{-1}\Vert }_{\infty }=O_P(1).$$ (42)

Substituting Equations (37), (39), (40), (41), and (42) into Equation (36), we find that

$${\Vert {\mathbb{E}}_n\left({\mathbf{ZZ}}_I^{\top }\right)\Vert }_{\infty }=O_P(n^{-1}{(log{p}_n)}^2),$$

which, by condition (3), satisfies Equation (34).

The order of magnitudes of the rest of the three terms can be derived similarly. We present the results below, omitting the details:

$$\begin{array}{l}{\Vert {\mathbb{E}}_n\left({\mathbf{ZZ}}_{II}^{\top }\right)\Vert }_{\infty }=O_P(n^{-1}log{p}_n),\\ {\Vert {\mathbb{E}}_n\left({\mathbf{ZZ}}_{\mathit{III}}^{\top }\right)\Vert }_{\infty }=O_P(n^{-1}{(log{p}_n)}^2),\\ {\Vert {\mathbb{E}}_n\left({\mathbf{Z}}_I{\mathbf{Z}}_I^{\top }\right)\Vert }_{\infty }=O_P(n^{-1}{(log{p}_n)}^2),\\ {\Vert {\mathbb{E}}_n\left({\mathbf{Z}}_I{\mathbf{Z}}_{II}^{\top }\right)\Vert }_{\infty }=O_P(n^{-1}log{p}_n),\\ {\Vert {\mathbb{E}}_n\left({\mathbf{Z}}_I{\mathbf{Z}}_{\mathit{III}}^{\top }\right)\Vert }_{\infty }=O_P(n^{-3/2}{(log{p}_n)}^2),\\ {\Vert {\mathbb{E}}_n\left({\mathbf{Z}}_{II}{\mathbf{Z}}_{II}^{\top }\right)\Vert }_{\infty }=O_P(n^{-1}),\\ {\Vert {\mathbb{E}}_n\left({\mathbf{Z}}_{II}{\mathbf{Z}}_{\mathit{III}}^{\top }\right)\Vert }_{\infty }=O_P(n^{-3/2}log{p}_n),\\ {\Vert {\mathbb{E}}_n\left({\mathbf{Z}}_{\mathit{III}}{\mathbf{Z}}_{\mathit{III}}^{\top }\right)\Vert }_{\infty }=O_P(n^{-1}{(log{p}_n)}^2).\end{array}$$

By condition (3), all of these terms satisfy the relation in Equation (34).

9. IMPLEMENTATION

In this section, we address two issues in implementation: the choice of the tuning parameter and the minimization of the objective functions (13) and (14). For the choice of the tuning parameter, we use a BIC-type criterion (Schwarz 1978) similar to that used in Yuan and Lin (2007). Let Θ̂(λ) = {θ̂_ij(λ) : i, j ∈ Γ} be the lasso or the adaptive lasso estimate of Θ₀ in the conditional graphical model for a specific choice of λ of the tuning parameter. Let E_n(λ) = {(i, j) : θ̂_ij(λ) ≠ 0}, and

$$\text{BIC}(\lambda )=-logdet[\widehat{\mathbf{\Theta }}(\lambda )]+\text{tr}[\widehat{\mathbf{\Theta }}(\lambda ){\mathbf{\sum }}_n]+logn\frac{\text{card}[E_n(\lambda )]+p}{2n}.$$

The tuning parameter is then chosen to be

$${\widehat{\lambda }}_{\text{BIC}}=\text{argmin}\{\text{BIC}(\lambda ):\lambda \in (0,\infty )\}.$$

Practically, we evaluate this criterion on a grid of points in (0, ∞) and choose the minimizer among these points.

For the minimization of Equations (13) and (14), we follow the graphical lasso procedure proposed by Friedman et al. (2008), but with the sample covariance matrix therein replaced by the RKHS estimates, Σ̂_PC(ε_n) or Σ̂_RR(ε_n), of the conditional covariance matrix Σ. The graphical lasso (glasso) procedure is available as a package in the R language.

10. SIMULATION STUDIES

In this section, we compare the sparse estimators for the CGGM with the sparse estimators of the GGM, with the maximum likelihood estimators of the CGGM, and with two naive estimators. We also explore several reproducing kernels and investigate their performances for estimating the CGGM.

10.1 Comparison With Estimators for GGM

We use three criteria for this comparison:

1. False positive rate at λ̂_BIC. This is defined as the percentage of edges identified as belonging to E when they are not; that is,

   $$\text{FP}=\text{card}\{(i,j):i>j,{\theta }_{0,ij}=0,{\widehat{\theta }}_{ij}\ne 0\}/\text{card}\{(i,j):i>j,{\theta }_{0,ij}=0\}.$$
2. False negative rate at λ̂_BIC. This is defined as the percentage of edges identified as belonging to E^c when they are not; that is,

   $$\text{FN}=\text{card}\{(i,j):i>j,{\theta }_{0,ij}\ne 0,{\widehat{\theta }}_{ij}=0\}/\text{card}\{(i,j):i>j,{\theta }_{0,ij}\ne 0\}.$$

3. Rate of correct paths. The above two criteria are both specific to the tuning method (in our case, BIC). To assess the potential capability of an estimator of E, independently of the tuning methods used, we use the percentage of cases where E belongs to the path {E_n(λ) : λ ∈ (0, ∞)}, where E_n(λ) is an estimator of E for a fixed λ. We write this criterion as PATH.

Example 1

This is the example, illustrated in Figure 1, in which p = 3, q = 1, and (X, Y) satisfies CGGM with E = {(1, 1), (2, 2), (3, 3), (2, 3), (3, 2)}. The conditional distribution of Y | X is specified by

$$\mathbf{Y}=\mathit{\beta }X+\mathit{\epsilon },$$ (43)

where β = (β₁, β₂, 0)^⊤, X ~ N(0, 1), and ε ~ N(0, Σ), X ⫫ ε, and

$${\mathbf{\sum }}^{-1}=\mathbf{\Theta }=\left(\begin{array}{ccc}5& 0& 0\\ 0& 4& 2.53\\ 0& 2.53& 2\end{array}\right).$$

For each simulated sample, β₁ and β₂ are generated, independently, from the uniform distribution defined on the set (−6, −3) ∪ (3, 6). We use two sample sizes n = 50, 100. The linear kernel κ_X(a, b) = 1 + a^⊤b is used for the initial RKHS estimate. The results are presented in the first three rows of Table 1. Entries in the table are the means calculated across 200 simulated samples.

Table 1.

Comparison of graph estimation accuracy among the lasso and the adaptive lasso estimators of GGM and CGGM for Example 1, Example 2 (including three scenarios), and Example 3. “ALASSO” means adaptive lasso

Example/scenario	Criteria	n = 50				n = 100
		LASSO		ALASSO		LASSO		ALASSO
		GGM	CGGM	GGM	CGGM	GGM	CGGM	GGM	CGGM
EX1	FP	1	0.51	1	0.15	1	0.40	1	0.05
	FN	0	0	0	0	0	0	0	0
	PATH	0	1	0	1	1	1	0	1
EX2-SC1	FP	0.84	0.02	0.54	0.06	0.93	0.01	0.67	0.02
	FN	0	0	0	0	0	0	0	0
	PATH	0	1	0.01	0.63	0	1	0.98	1
EX2-SC2	FP	0.98	0.55	0.96	0.31	1	0.51	1	0.22
	FN	0.06	0.01	0.15	0.02	0.02	0	0.08	0
	PATH	0	0.57	0	0.71	0	0.79	0	0.98
EX2-SC3	FP	0.71	0.68	0.18	0.19	0.75	0.77	0.10	0.11
	FN	0	0	0.01	0.02	0	0	0	0
	PATH	0	0.43	0.80	0.52	0	0.56	1	0.87
EX3	FP	0.79	0.23	0.41	0.09	0.83	0.16	0.59	0.03
	FN	0	0	0	0	0	0	0	0
	PATH	0	0.94	0.95	0.99	0	1	1	1

The table indicates that the unconditional sparse estimators have much higher false positive rate than false negative rate. This is because they tend to pick up connections among the components of Y that are due to X. In comparison, the conditional sparse estimators (both lasso and adaptive lasso) can successfully remove the edges effected by X, resulting in more accurate identification of the graph.

Example 2

In this example, we consider three scenarios in which the effect of an external source on the network varies in degree, resulting in different amounts of gain achievable by a conditional graphical model.

We still assume the linear regression model (43), but with p = 5. The first scenario is shown in the top two panels in Figure 2, where all the components of β are nonzero and Σ diagonal. In this case, the conditional graph is totally disconnected (left panel); whereas the unconditional graph is a complete graph (right panel). Specifically, the parameters are

Figure 2.

Conditional and unconditional graphical models in Examples 2 and 3. Left panels: the conditional graphical models. Right panels: the corresponding unconditional graphical models. Black nodes indicate response variables; gray nodes are the predictors. Edges in the conditional and unconditional graphs are indicated by black lines; regression of Y on X is indicated by directed gray lines with arrows. (The online version of this figure is in color.)

$$\mathbf{\sum }=\mathbf{\Theta }={\mathbf{I}}_5,\mathit{\beta }={(0.656,0.551,0.757,0.720,0.801)}^{\top }.$$

The panels in the third row of Figure 2 represent the other extreme, where only Y¹ is related to X, and each Yⁱ for i ≠ 1 shares an edge with Y¹ but has no other edges. In this case, the conditional and unconditional graphs are identical. The parameters are specified as follows. The first row (and first column) of Θ is (6.020, −0.827, −1.443, −1.186, −0.922); the remaining 4 × 4 block is I₄. The first entry of β is 0.656, and rest entries are 0. Between these two extremes is scenario 2 (second row in Figure 2), where the conditional and unconditional graphs differ only by one edge: 1 ↔ 4. The parameters are

$$\begin{array}{l}\mathbf{\Theta }=\left(\begin{array}{ccccc}4.487& -1.186& -1.443& 0& 0\\ -1.186& 1.464& -0.668& 0.752& -0.681\\ -1.443& -0.668& 1.963& -1.084& 0.981\\ 0& 0.752& -1.084& 2.220& -1.105\\ 0& -0.681& 0.981& -1.105& 1\end{array}\right),\\ \mathit{\beta }=\left(\begin{array}{c}0.656\\ 0.551\\ 0\\ 0.601\\ 0\end{array}\right).\end{array}$$

For each scenario, we generate 200 samples of sizes n = 50, 100 and compute the three criteria across the 200 samples. The results are presented row 4 through row 12 of Table 1. We see significant improvements of the rates of correct identification of the graphical structure by the sparse estimator of CGGM whenever the conditional graph differs from the unconditional graph. Also note that, for scenario 3, where the two graphs are the same, the adaptive lasso estimator for GGM performs better than the adaptive lasso estimator for CGGM. This is because the latter needs to estimate more parameters.

Example 3

In this example, we investigate a situation where the edges in the unconditional graphical model have two external sources. We use model (43) with p = 5, q = 2, X ~ N(0, I₂),

$$\begin{array}{l}\mathbf{\Theta }=\left(\begin{array}{ccccc}1.683& -0.827& 0& 0& 0\\ -0.827& 1& 0& 0& 0\\ 0& 0& 1.722& 0.849& 0\\ 0& 0& 0.849& 1& 0\\ 0& 0& 0& 0& 1\end{array}\right),\\ \beta =\left(\begin{array}{cc}0.656& 0\\ 0& 0.551\\ 0.757& 0\\ 0& 0.720\\ 0& 0.800\end{array}\right).\end{array}$$

The results are summarized in the last three rows of Table 1, from which we can see similar improvements by the sparse CGGM estimators.

10.2 Comparison With Maximum Likelihood Estimates of CGGM

In Section 7, we showed that the adaptive lasso estimate for the CGGM possesses oracle property; that is, its asymptotic variance reaches the lower bound among regular estimators when the graph is assumed known. Hence, it makes sense to compare adaptive lasso estimate with the maximum likelihood estimate of Θ under the constraints θ_ij = 0, (i, j) ∉ E, which is known to be optimal among regular estimates. In this section, we make such a comparison, using all three examples in Section 10.1. As a benchmark, we also compare these two estimates with the maximum likelihood estimate under the full model, which for the linear kernel is [Σ̂_PC(0)]^†. For this comparison we use the squared Frobenius norm ${\Vert \widehat{\mathbf{\Theta }}-{\mathbf{\Theta }}_0\Vert }_F^2$, which characterizes the closeness of two precision matrices rather than that of graphs.

Table 2 shows that the adaptive lasso estimator is rather close to the constrained MLE, with the unconstrained MLE trailing noticeably behind. In three out of five cases, the constrained MLE performs better than the adaptive lasso, which is not surprising because, although the two estimators are equivalent asymptotically, the former employs the true graphical structure unavailable for adaptive lasso, making it more accurate for the finite sample. When the errors of adaptive lasso are lower than the constrained MLE, the differences are within the margins of error.

Table 2.

Comparison of parameter estimation accuracy among adaptive lasso, and unconstrained and constrained MLE for CGGM. Entries are of the form a ± b, where a is the mean, and b the standard deviation, of criterion ${\Vert \widehat{\mathbf{\Theta }}-{\mathbf{\Theta }}_0\Vert }_F^2$ computed from 200 simulated samples

Example/scenario	ALASSO	MLE
		Unconstrained	Constrained
EX1	1.458 ± 0.125	2.256 ± 0.171	1.575 ± 0.155
EX2-SC1	0.142 ± 0.008	0.400 ± 0.018	0.122 ± 0.007
EX2-SC2	1.858 ± 0.120	2.294 ± 0.148	1.663 ± 0.116
EX2-SC3	1.147 ± 0.071	2.139 ± 0.186	0.969 ± 0.081
EX3	0.303 ± 0.016	0.773 ± 0.555	0.327 ± 0.275

10.3 Exploring Different Reproducing Kernels

In this section, we explore three types of kernels for RKHS

$$\begin{array}{l}\text{polynomial}(\text{PN}):{\kappa }_X(\mathbf{a},\mathbf{b})={({\mathbf{a}}^{\top }\mathbf{b}+1)}^r,\hfill \\ \text{Gauss}\text{radial}\text{basis}(\text{RB}):{\kappa }_X(\mathbf{a},\mathbf{b})=exp(-\gamma {\Vert \mathbf{a}-\mathbf{b}\Vert }^2),\hfill \\ \text{rational}\text{quadratic}(\mathbf{RQ}):{\kappa }_X(\mathbf{a},\mathbf{b})=1-{\Vert \mathbf{a}-\mathbf{b}\Vert }^2/\times ({\Vert \mathbf{a}-\mathbf{b}\Vert }^2+c),\hfill \end{array}$$ (44)

and investigate their performances as initial estimates for lasso and adaptive lasso. These kernels are widely used for RKHS (Genton 2001). For the CGGM, we use a nonlinear regression model with four combinations of dimensions: q = 10, 20, p = 50, 100. The nonlinear regression model is specified by

$$Y^i=\{\begin{array}{ll}{\left({\mathit{\beta }}_1^{\top }\mathbf{X}+1\right)}^2+{\epsilon }^i,\hfill & i=1,3,6,8,\dots ,p-4,p-2,\hfill \\ {\left({\mathit{\beta }}_2^{\top }\mathbf{X}\right)}^2+{\epsilon }^i,\hfill & i=2,4,5,7,9,10,\dots ,\hfill \\ \hfill & p-3,p-1,p,\hfill \end{array}$$ (45)

where β₁ and β₂ are q-dimensional vectors

$$\begin{array}{l}{\mathit{\beta }}_1={(\underset{q/2}{\underbrace{1,\dots ,1}},\underset{q/2}{\underbrace{0,\dots ,0}})}^{\top },\\ {\mathit{\beta }}_2={(\underset{q/2}{\underbrace{0,\dots ,0}},\underset{q/2}{\underbrace{1,-1,\dots ,{(-1)}^{q/2+1}}})}^{\top }.\end{array}$$

The distribution of (ε¹, …, ε^p)^⊤ is multivariate normal with mean 0 and precision matrix

$$\left(\begin{array}{ccc}\mathbf{\Gamma }& & 0\\ & \ddots & \\ 0& & \mathbf{\Gamma }\end{array}\right)$$

where each Γ is the precision matrix in Example 3.

The following specifications apply throughout the rest of Section 10: γ = 1/(9q) for RB, c = 200 for RQ, and r = 2 for PN (because the predictors in Equation (45) are quadratic polynomials); the RKHS estimator Σ̂_PC(ε_n) is used as the initial estimator for lasso and adaptive lasso, where ε_n are chosen so that the first 70 eigenvectors of QK_XQ are retained. Ideally, the kernel parameters γ, c, and ε_n should be chosen by data-driven methods such as cross-validation. However, this is beyond the scope of the present article and will be further developed in a future study. Our choices are based on trial and error in pilot runs. Our experience indicates that the sparse estimators perform well and are reasonably stable when ε_n is chosen so that 10% ~ 30% of the eigenvectors of QK_XQ are included. The sample size is n = 100 and the simulation is based on 200 samples. To save computing time we use the BIC to optimize λ_n for the first sample and use it for the rest 199 samples.

In Table 3, we compare the sparse estimators lasso and adaptive lasso, whose initial estimates are derived from kernels in Equation (44), with the full and constrained MLEs. The full MLE is computed using the knowledge that the predictor is a quadratic polynomial of x₁, …, x_p. The constrained MLE uses, in addition, the knowledge of the conditional graph ; that is, the positions of the zero entries of the true conditional precision matrix.

Table 3.

Exploration of different reproducing kernels. Entries are ${\Vert \widehat{\mathbf{\Theta }}-{\mathbf{\Theta }}_0\Vert }_F^2$ averaged over 200 simulation samples

p	q	LASSO			ALASSO			MLE
		PN	RB	RQ	PN	RB	RQ	Full	Constrained
50	10	1.84	6.88	8.14	1.84	2.31	2.71	29.41	3.82
	20	11.03	11.03	11.03	17.15	17.14	17.14	299.28	94.18
100	10	5.96	20.07	24.06	3.35	4.33	5.17	171.32	7.73
	20	20.06	20.05	20.05	29.10	28.30	28.31	1787.32	186.39

Table 3 shows that in all cases the sparse estimators perform substantially better than the full MLEs. For q = 10, the adaptive lasso estimates based on all three kernels also perform better than both the full and constrained MLEs; whereas the accuracy of most of the lasso estimates are between the full and the constrained MLEs. For q = 20, all sparse estimators perform substantially better than both the full and the constrained MLEs. From these results, we can see the effects of two types of regularization: the sparse regularization of the conditional precision matrix and the kernel-PCA regularization for the predictor. The first regularization counteracts the increase in the number of parameters in the conditional precision matrix as p increases, and the second counteracts the increase in the number of terms in a quadratic polynomial as q increases, both resulting in substantially reduced estimation error.

10.4 Comparisons With Two Naive Estimators

We now compare our sparse RKHS estimators for the CGGM with two simple methods: the linear regression and the simple thresholding.

10.4.1 Naive Linear Regression

A simple estimate of CGGM is to first apply multivariate linear regression of Y versus X, regardless of the true regression relation, and then apply a sparse penalty to the residual variance matrix. To make a fair comparison with linear regression, we consider the following class of regression models:

$$\mathbf{Y}=(1-a){({\mathit{\beta }}^{\top }\mathbf{X})}^2/4+a({\mathit{\beta }}^{\top }\mathbf{X})+\mathit{\epsilon },0\le a\le 1.$$ (46)

This is a convex combination of a linear model and a quadratic model: it is linear when a = 1, quadratic when a = 0, and a mixture of both when 0 < a < 1. The distribution of ε is as specified in Section 10.3. The fraction 1/4 in the quadratic term in Equation (46) is introduced so that the linear and quadratic terms have the similar signal-to-noise ratios. In Table 4, we compare the estimation error of the CGGM based on Equation (46) by sparse linear regression and by sparse RKHS estimator using the adaptive lasso as penalty. We take q = 10, p = 50, and n = 500. The Gauss radial basis is used for the kernel method, with tuning parameters γ and ε_n being the same as specified in Section 10.3.

Table 4.

Comparison with linear regression. Entries are ${\Vert \widehat{\mathbf{\Theta }}-{\mathbf{\Theta }}_0\Vert }_F^2$ averaged over 200 samples

a	0	0.2	0.4	0.6	0.8	1
Linear	10.49	10.48	10.48	10.42	10.10	0.75
Kernel	1.56	2.13	1.77	1.72	1.67	1.56

We see that the sparse RKHS estimate performs substantially better in all cases except a = 1, where Equation (46) is exactly a linear model. This suggests that linear-regression sparse estimate of CGGM is rather sensitive to nonlinearity: a slight proportion of nonlinearity in the mixture would make the kernel method favorable. In comparison, the sparse RKHS estimate is stable and accurate for different types of regression relations.

10.4.2 Simple Thresholding

A naive approach to sparsity is by dropping small entries of a matrix. For example, we can estimate Θ by setting to zero the entries of ${\widehat{\mathbf{\sum }}}_{\mathbf{PC}}^{-1}$ whose absolute values are smaller than some τ > 0. Let Θ̃(τ) denote this estimator. Using the example in Section 10.3 with p = 50, q = 10, n = 500, we now compare the sparse RKHS estimators with Θ̃(τ). The curve in Figure 3 is the error ${\Vert \stackrel{\sim }{\mathbf{\Theta }}(\tau )-\mathbf{\Theta }\Vert }_F^2$ versus τ ∈ [0, 1]. Each point in the curve is the error over 200 simulated samples. The estimate Σ̂_PC is based on the gauss radial basis, whose tuning parameters γ and ε_n are as given in Section 10.3. The two horizontal lines in Figure 3 represent the errors for the lasso and the adaptive lasso estimators based on Σ̂_PC, which are read off from Table 3.

Figure 3.

Comparison of lasso, adaptive lasso, and simple thresholding.

The figure shows that the simple thresholding estimate, even for the best threshold, does not perform as well as either of our sparse estimates. Note that in practice Θ₀ is unknown, and the optimal τ cannot be obtained by minimizing the curve in Figure 3. With this in mind, we expect the actual gap between the thresholding estimate and the sparse estimates to be even greater than that shown in the figure.

11. NETWORK INFERENCE FROM eQTL DATA

In this section, we apply our CGGM sparse estimators to two datasets to infer gene networks from expression quantitative trait loci (eQTL) data. The dataset is collected from an F2 intercross between inbred lines C3H/HeJ and C57BL/6J (Ghazalpour et al. 2006). It contains 3,421 transcripts and 1,065 markers from the liver tissues of 135 female mice (n = 135). The purpose of our analysis is to identify direct gene interactions by fitting the CGGM to the eQTL dataset. Although a gene network can be inferred from expression data alone, such a network would contain edges due to confounders such as shared genetic causal variants. The available marker data in the eQTL dataset allow us to isolate the confounded edges by conditioning on the genomic information.

We restrict our attention to subsets of genes, partly to accommodate the small sample size. In the eQTL analysis tradition, subsets of genes can be identified by two methods: co mapping and co expression. For co-mapping, each gene expression trait is mapped to markers, and the transcripts that are mapped to the same locus are grouped together. For co-expression, the highly correlated gene expressions are grouped together.

We first consider the subset of genes identified by co-mapping. It has been reported (Neto, Keller, Attie, and Yandell 2010) that 14 transcripts are mapped to a marker on chromosome 2 (at 55.95 cM). As this locus is linked to many transcripts, it is called a hot-spot locus. It is evident that this marker should be included as a covariate for CGGM. In addition, we include a marker on chromosome 15 (at 76.65 cM) as a covariate, because it is significantly linked to gene Apbb1ip (permutation p-value < 0.005), conditioning on the effect of the marker on chromosome 2. The transcript mapping is performed using the qtl package in R (Broman, Wu, Sen, and Churchill 2003).

The GGM detects 52 edges; the CGGM detects 34 edges, all in the set detected by the GGM. The two graphs are presented in the upper panel of Figure 4, where edges detected by both GGM and CGGM are represented by black solid lines, and edges detected by GGM alone are represented by blue dotted lines. In the left panel of Table 5, we compare the connectivity of genes of each graph. Among the 14 transcripts, Pscdbp contains the hot spot locus within the ±100 kb boundaries of its location. Interestingly, in CGGM, this cis transcript has the lowest connectivity among the 14 genes, but one of the genes with which it is associated (Apbb1ip) is a hub gene, connected with seven other genes. In sharp contrast, GGM shows high connectivity of Pscdbp itself.

Figure 4.

Gene networks based on GGM and CGGM. Upper panel: data based on co mapping selection. Lower panel: data based on coexpression selection. An edge from both GGM and CGGM is represented by a solid line; an edge from the GGM alone is represented by a dotted line; an edge from CGGM alone is represented by a broken line. (The online version of this figure is in color.)

Table 5.

Connectivity of genes in the conditional and unconditional graphical models. The cis-regulated transcripts are indicated with boldface

Co-mapping				Co-expression
Gene	GGM	CGGM	Diff.	Gene	GGM	CGGM	Diff.
Il16	7	3	4	Aqr	7	6	1
Frzb	7	5	2	Nola3	8	2	6
Apbb1ip	8	7	1	AI648866	9	6	3
Clec2	6	4	2	Zfp661	8	5	3
Riken	6	4	2	Myef2	11	5	6
Il10rb	8	5	3	Synpo2	9	3	6
Myo1f	8	5	3	Slc24a5	6	5	1
Aif1	6	5	1	Nol5a	7	4	3
Unc5a	11	8	3	Sdh1	10	6	4
Trpv2	9	6	3	Dtwd1	9	4	5
Tnfsf6	9	4	5	B2m	8	6	2
Stat4	3	3	0	Tmem87a	6	3	3
Pscdbp	9	3	6	Zc3hdc8	6	5	1
D13Ertd275e	7	6	1

We next study the subset identified by co-expression. We use a hierarchical clustering approach in conjunction with the average agglomeration procedure to partition the transcripts into 10 groups. The relevant dissimilarity measure is 1 − |ρ_ij|, where ρ_ij is the Pearson correlation between transcripts i and j. Among the 10 groups, we choose a group that contains 15 transcripts with the mean absolute correlation equal to 0.78. Thirteen of the 15 transcripts have annotations, and they are used in our analysis. Using the qtl package in R, each transcript is mapped to markers, and seven markers on chromosome 2 are significantly linked to transcripts (permutation p-value 0.005). Among those, we drop two markers that are identical to the adjacent markers. We thus use five markers (Chr2@100.18, Chr2@112.75, Chr2@115.95, Chr2@120.72, Chr2@124.12) as covariates.

The GGM identifies 52 edges and the CGGM identifies 30 edges. Among these edges, 28 are shared by both methods. The two graphs are presented in the lower panel of Figure 4, where edges detected by both GGM and CGGM are represented by black solid lines, edges detected by GGM alone are represented by blue dotted lines, and edges detected by CGGM alone are represented by red broken lines. Among the 13 transcripts, Dtwd1 is the closest to all markers (distances < 300 kbp). The connectivity of each method is shown in the right panel of Table 5. The number of genes that are connected to Dtwd1 is reduced by 5 by CGGM. Unlike in the co-mapping network, in the co-expression network, CGGM detects two edges (Aqr–Zfp661, Zfp661–Tmem87a) that are not detected by GGM. These additional edges could be caused by the error in regression estimation. For example, if the regression coefficients for some of markers are 0, but are estimated to be nonzero, then spurious correlations arise among residuals. This indicates that the accurate identification of the correct covariates is more important in the co-expression network.

As a summary, in the network inference from the transcripts identified by co-mapping, we see that after conditioning on the markers, the cis-regulated transcripts (Pscdbp) are connected to relatively few genes of high connectivity. However, without conditioning on the markers, they appear to have high connectivity themselves. In other words, without conditioning on markers, these cis-regulated transcripts might be misinterpreted as hub-genes themselves. In the network inference from the transcripts identified by co expression, we see that after conditioning on the markers, a few edges are additionally detected, which may result from inaccurate identification of covariates for an individual transcript. Thus, including the correct set of markers for an individual transcript can be important.