On an Additive Semigraphoid Model for Statistical Networks With Application to Pathway Analysis

Abstract

We introduce a nonparametric method for estimating non-gaussian graphical models based on a new statistical relation called additive conditional independence, which is a three-way relation among random vectors that resembles the logical structure of conditional independence. Additive conditional independence allows us to use one-dimensional kernel regardless of the dimension of the graph, which not only avoids the curse of dimensionality but also simplifies computation. It also gives rise to a parallel structure to the gaussian graphical model that replaces the precision matrix by an additive precision operator. The estimators derived from additive conditional independence cover the recently introduced nonparanormal graphical model as a special case, but outperform it when the gaussian copula assumption is violated. We compare the new method with existing ones by simulations and in genetic pathway analysis.

1 Introduction

The gaussian graphical model is one of the most important tools for statistical network analysis, partly because its estimation can be neatly formulated as identifying the zero entries in the precision matrix, which turns estimation of a graph into sparse estimation of a high-dimensional precision matrix. For recent advances in this area, see Meinshausen and Buhlmann (2006), Yuan and Lin (2007), Bickel and Levina (2008), Peng, Wang, Zhou, and Zhu (2009), Guo, Levina, Michailidis, and Zhu (2009), Lam and Fan (2009).

Specifically, let X = (X¹, …, X^p)^⊤ be a p-dimensional random vector distributed as N(μ, Σ), where Σ is positive definite. Let Γ = {1, …, p}, E ⊆ {(i, j) ∈ Γ × Γ : i ≠ j}, and Θ = Σ⁻¹. Let θ_ij be the (i, j)th element of Θ. Then X follows a gaussian graphical model (GGM) with respect to the graph 𝓖 = {Γ, E} iff

$$X^i⫫X^j\mid X^{-(i,j)},(i,j)\notin E,$$ (1)

where X^−(i,j) represents the set {X^k : k ≠ i, j}. Under the multivariate gaussian assumption, the above relation is equivalent to

$${\theta }_{ij}=0,(i,j)\notin E.$$ (2)

Based on this equivalence the estimation of 𝓖 reduces to identification of the 0 entries in Θ, which can be done by sparse estimation procedures such as thresholding (Bickel and Levina, 2008), LASSO (Tibshirani, 1996), SCAD (Fan and Li, 2001), and adaptive LASSO (Zuo, 2006).

Since the multivariate gaussian assumption limits the applicability of the GGM, there have been intense developments in generalizing it to non-gaussian cases in the past few years, chiefly through the use of nonparametric copula transformations. See Liu, Lafferty, and Wasserman (2009), Liu, Han, Yuan, Lafferty, and Wasserman (2013), and Xue and Zou (2013). The nonparametric gaussian copula graphical model (GCGM) is a flexible and effective model that relaxes the marginal gaussian assumption but preserves the inter-dependence structure of a multivariate gaussian distribution, so that the equivalence between (1) and (2) still holds in an altered form. It assumes the existence of unknown injections f₁, …, f_p such that (f₁(X¹), …, f_p(X^p))^⊤ is multivariate gaussian. The GCGM keeps the simplicity and elegance of the GGM but greatly expands its scope.

However, the gaussian copula assumption can be violated even for commonly used interactions. For example, if (U, V) is a random vector where V = U² + ε, and U and ε are i.i.d. N(0, 1), then no transformation can render (f₁(U), f₂(V)) bivariate gaussian. In fact, the copula transformations in this case are: f₁(u) = u, f₂(v) = Φ⁻¹∘F_V(v), where F_V is the convolution of N(0, 1) and ${\chi }_1^2$ and Φ is the c.d.f. of N(0, 1). Figure 1 plots (U, Φ⁻¹∘F_V(V)) for a sample of 100 observed pairs, which shows a strong non-gaussian pattern even though U and Φ⁻¹∘F_V(V) are marginally gaussian. Under such circumstances the gaussian copula assumption no longer holds, and our goal is to develop a broader class of graphical models that cover such cases.

Figure 1.

Violation of copula assumption. The vertical axis is Φ⁻¹∘F_V(V). Both U and Φ⁻¹∘F_V(V) are N (0, 1) but their joint distribution is strongly non-gaussian.

In developing the new graphical models we would like to preserve an appealing feature of the GCGM; that is, the nonparametric operation is acted upon one-dimensional random variables individually, rather than a high-dimensional random vector jointly. This feature allows us to avoid high-dimensional smoothing, which is the source of the curse of dimensionality (Bellman, 1957). However, if we insist on using conditional independence (1) as the criterion for constructing graphical models, then we are inevitably lead to a fully fledged nonparametric procedure involving smoothing over the entire p-dimensional space, and the nice structures of the gaussian graphical model such as (2) would be lost. This motivates us to introduce a new statistical relation — additive conditional independence, or ACI, to replace conditional independence as our criterion. Although ACI is different from conditional independence, it satisfies the axioms for a semi-graphoid (Pearl and Verma, 1987) shared by conditional independence, which captures the essence of a graph. Furthermore, under the gaussian copula assumption, ACI reduces to conditional independence. The graphical model defined by ACI — which we refer to as the Additive Semi-graphoid Model (ASG) — preserves a parallel structure to (2

The paper is organized follows. In section 2 we outline the axiomatic system for a semi-graphoid logical relation, and introduce the notion of ACI as well as the ASG model it induces. In section 3 we investigate the relation between ACI and conditional independence under the gaussian copula assumption. In sections 4 and 5, we introduce the additive conditional covariance operator and additive precision operator, and study their relations with ACI. In sections 6 and 7 we introduce two estimators for the ASG model based on these operators. In section 8 we propose cross validation procedures to determine the tuning parameters. In sections 9 and 10 we compare the ASG model with GGM and GCGM by simulation and in an applied setting of genetic pathway inference. We conclude in section 11 with suggested future research directions. All proofs are presented in the Appendix.

2 Semi-graphoid axioms and additive conditional independence

Before introducing additive conditional independence, we first restate the axiomatic system developed by Pearl and Verma (1987) and Pearl, Geiger, and Verma (1989), in a slightly different form. Let 2^Γ be the class of all subsets of Γ. We call any subset ℛ ⊆ 2^Γ × 2^Γ × 2^Γ a three-way relation on Γ. This terminology follows the tradition of two-way relations in mathematics (see, for example, Kelley, 1955, page 6).

Definition 1 (Semi-graphoid Axioms)

A three-way relation ℛ is called a semi-graphoid if it satisfies the following conditions:

1. (symmetry) (A, C, B) ∈ ℛ ⇒ (B, C, A) ∈ ℛ;

2. (decomposition) (A, C, B ∪ D) ∈ ℛ ⇒ (A, C, B) ∈ ℛ;

3. (weak union) (A, C, B ∪ D) ∈ ℛ ⇒ (A, C ∪ B, D) ∈ ℛ;

4. (contraction) (A, C ∪ B, D) ∈ ℛ, (A, C, B) ∈ ℛ ⇒ (A, C, B ∪ D) ∈ ℛ.

These axioms are extracted from conditional independence to convey the general idea of “B is irrelevant for understanding A once C is known”, or “C separates A and B”. Conditional independence is a special case of the semi-graphoid three-way relation. More specifically, let X = {Xⁱ : i ∈ Γ} be a random vector and, for any A ∈ Γ, let X^A be the {Xⁱ : i ∈ A}. Let ℛ be the class of (A, C, B) such that X^A ⫫ X^B|X^C (X^A and X^B are conditionally independent given X^C), then it can be shown that ℛ indeed satisfies the semi-graphoid axioms (Dawid, 1979).

Because, like conditional independence, a semi-graphoid relation (A, C, B) ∈ ℛ also conveys the idea that C separates A and B, it seems reasonable to use the semi-graphoid relation itself as a criterion for constructing a graph. Indeed, the very statement “there is no direct link between node i and node j” means, precisely, that “{i, j}^c separates i and j”. In other words, it is reasonable to use the logical statement ({i}, {i, j}^c, {j}) ∈ ℛ to determine the absence of edge between i and j. Using the semi-graphoid relation in place of probabilistic conditional independence can free us from some awkward restrictions that come with a fully fledged probability model.

Let X = (X¹, …, X^p)^⊤ be a random vector. For any subvector U = (U¹, …, U^r)^⊤ of X, let L₂(P_U) be the class of functions of U such that Ef(U) = 0, Ef²(U) < ∞. For each Uⁱ, let 𝓐_Uⁱ denote a subset of L₂(P_Uⁱ). Let 𝓐_U denote the additive family

$${\mathcal{A}}_{U^1}+\cdots +{\mathcal{A}}_{U^r}=\{f_1+\cdots +f_r:f_1\in {\mathcal{A}}_{U^1},\dots ,f_r\in {\mathcal{A}}_{U^r}\}.$$

For two subspaces 𝓐 and 𝓑 of L₂(P_X), let 𝓐 ⊖ 𝓑 = 𝓐 ∩ 𝓑^⊥, where orthogonality is in terms of the L₂(P_X)-inner product. In the following, an unadorned inner product 〈·, ·〉 always represents the L₂(P_X)-inner product.

Definition 2

Let U, V, and W be subvectors of X. We say that U and V are additively conditionally independent (ACI) given W iff

$$({\mathcal{A}}_U+{\mathcal{A}}_W)\ominus {\mathcal{A}}_W\perp ({\mathcal{A}}_V+{\mathcal{A}}_W)\ominus {\mathcal{A}}_W,$$ (3)

where ⊥ indicates orthogonality in the L₂(P_X)-inner product. We write relation (3) as U ⫫_AV|W.

Strictly speaking, ACI should be stated with respect to the triplet of functional families (𝓐_U, 𝓐_V, 𝓐_W), but in most cases we can ignore this qualification without loss of clarity. Lauritzen (1996, page 31) mentioned a relation similar to (3) where 𝓐_U, 𝓐_V, and 𝓐_W are Euclidean subspaces, but absent from that construction are the underlying random vectors U, V, W, which is crucial to our development.

The next theorem gives seven equivalent conditions for ACI, each of which can be used as its definition. As we will see in the proof of Theorem 2, these various forms are immensely useful in exploring the ramifications of ACI. In the following, for an operator P on a Hilbert space 𝓗, let ker P and ranP denote the kernel and range of P, and let $\overline{\text{ran}}P$ be the closure of ranP.

Theorem 1

Let U, V, W be subvectors of X. The following statements are equivalent:

1. (𝓐_U + 𝓐_W) ⊖ 𝓐_W ⊥ (𝓐_V + 𝓐_W) ⊖ 𝓐_W;

2. 𝓐_U∪W ⊖ 𝓐_W ⊥ 𝓐_V∪W ⊖ 𝓐_W;

3. (I − P_𝓐W) 𝓐_U ⊥ (I − P_𝓐W)𝓐_V;

4. 𝓐_U ⊆ ker(P_{𝓐V∪W ⊖ 𝓐W});

5. 𝓐_U ⊆ ker(P_𝓐V∪W − P_𝓐W);

6. 𝓐_U∪W ⊆ ker(P_{𝓐V∪W ⊖ 𝓐W});

7. 𝓐_U∪W ⊆ ker(P_𝓐V∪W − P_𝓐W).

The next theorem shows that ACI satisfies the semi-graphoid axioms, which gives justification of using it to construct graphical models.

Theorem 2

The additive conditional independence relation is a semi-graphoid.

We are now ready to define the additive semi-graphoid model.

Definition 3

A random vector X follows an additive semi-graphoid model with respect to a graph 𝓖 = (Γ, E) iff

$$X^i{⫫}_A{X}^j\mid X^{-(i,j)}\iff (i,j)\notin E.$$ (4)

We write this condition as X ~ ASG(𝓖).

3 Relation with copula graphical models

While in the last section we have seen it is reasonable to use ACI instead of conditional independence as a criterion for constructing graphs, we now investigate the special cases where ACI reduces conditional independence.

Theorem 3

Suppose

1. X has a gaussian copula distribution with copula functions f₁, …, f_p;

2. U, V, and W are subvectors of X;

3. 𝓐_Xⁱ = span{f_i}, i = 1, …, p.

Then U ⫫_AV|W if and only if U ⫫V|W.

Under the specific form of ASG model in Theorem 3, there is no edge between i and j iff the (i, j)th entry of the precision matrix of (f₁(X¹), …, f_p(X^p))^⊤ is 0. The latter is population-level criterion used by the GCGM estimator. In this sense, ASG can be regarded as a generalization of the nonparanormal models.

Since our main interests are in the ASG models with 𝓐_Xⁱ = L₂(P_Xⁱ), it is also of interest to ask whether ⫫_A coincides with ⫫ in this case. The next theorem and the subsequent numerical investigation show that they are very nearly equivalent for all practical purposes.

Theorem 4

Suppose

1. X has a gaussian copula distribution with copula functions f₁, …, f_p;

2. U, V, and W are sub-vectors of X;

   3. 𝓐_Xⁱ = L₂(P_Xⁱ).

Then U ⫫_AV|W implies U ⫫ V|W.

The proof, which is omitted, is essentially the same as that of Theorem 3. Interestingly, under 𝓐_Xⁱ = L₂(P_Xⁱ), ⫫ does not imply ⫫_A even under the gaussian copula assumption. However, the following numerical investigation suggests that this implication holds approximately, with vanishingly small error. We are able to carry out this investigation because an upper bound of

$${\rho }_{UV\mid W}^2=sup\{{\langle f-P_{{\mathcal{A}}_W}f,g-P_{{\mathcal{A}}_W}g\rangle }^2:f\in L_2(P_U),g\in L_2(P_V),\Vert f\Vert =\Vert g\Vert =1\}$$

can be calculated explicitly under the copula gaussian distribution. To our knowledge this bound has not been recorded in the literature; so it is of interest in its own.

Proposition 1

Suppose X has a gaussian copula distribution with copula functions f₁, …, f_p. Without loss of generality, suppose E[f_i(Xⁱ)] = 0, var[f_i(Xⁱ)] = 1. Let

$$U=f_i(X^i),V=f_j(X^j),W=\{f_k{(X)}^k:k\ne i,j\},$$

and R_VW = cov(V, W), R_WW = var(W), R_WU = cov(W, U), ρ_UV = cor(U, V). Then

$${\rho }_{UV\mid W}\le max\{\mid {\rho }_{UV}^{\alpha }-R_{VW}^{\odot \alpha }{(R_{WW}^{\odot \alpha })}^{-1}{R}_{WU}^{\odot \alpha }\mid :\alpha =1,2,\dots \}\equiv {\tau }_{UV\mid W},$$ (5)

A^⊙α is the α-fold Hadamard product A ⊙ ··· ⊙ A and |·| means determinant.

Using this proposition we conduct the following numerical investigation. First we generate a positive definite random matrix C with C₁₂ = C₂₁ = 0 as follows. We generate C̃ from the Wishart distribution W(I_p, k), where k is the degrees of freedom, and then set C̃₁₂ and C̃₂₁ to 0. This does not guarantee C̃ to be positive definite. If it is not, we repeat this process until we get a positive definite matrix, which is then set to be C. Let R_XX = [diag(C)]^1/2C⁻¹[diag(C)]^1/2. Let R_WU, R_WV, and R_WW be the (p − 2) × 1, (p − 2) × 1, and (p − 2) × (p − 2) matrices

$$\{{(R_{XX})}_{i1}:i=3,\dots ,p\},\{{(R_{XX})}_{i2}:i=3,\dots ,p\},\{{(R_{XX})}_{ij}:i,j=3,\dots ,p\},$$

and let $R_{UW}=R_{WU}^{\top },R_{VW}=R_{WV}^{\top }$. Note that, if X ~ N (0, R_XX), then it is guaranteed that U ⫫ V|W, and var(Xⁱ) = 1.

We choose the degrees of freedom k to be relatively small to prevent C/k from converging to I_p, but large enough to avoid singularity. In particular, we take k = ⌈5p/4⌉, ⌈6p/4⌉, …, ⌈8p/4⌉, which guarantees a wide range of covariance matrices. We take p = 20, 40, …, 100. We compute τ_UV|W by taking the maximum of the first 10 terms in (5), which in our simulations is always the global maximum. For each combination of (p, k), we generate 500 random matrices and present the averages of τ_UV|W in Table 1. Because ρ_UV|W = 0 iff U ⫫_AV|W, the small numbers in Table 1 indicate that for all practical purposes we can treat U ⫫ V|W ⇒ U ⫫_AV|W as valid under the gaussian copula model, even though it is not a mathematical fact.

Table 1.

Average values of τ_UV|W under U ⫫ V|W

p	k
	⌈5p/4⌉	⌈6p/4⌉	⌈7p/4⌉	⌈8p/4⌉
20	0.00310	0.00270	0.00114	0.00087
40	0.00472	0.00240	0.00095	0.00051
60	0.00485	0.00185	0.00069	0.00024
80	0.00435	0.00183	0.00053	0.00018
100	0.00483	0.00125	0.00036	0.00018

In the similar way it is related to the gaussian copula graphical models, the ASG is also related to the more general transelliptical graphical models recently introduced by Liu, Han, and Zhang (2012). A random vector X is said to have a transelliptical distribution with shape parameter Σ (a p × p positive definite matrix) if there exist injections f₁, …, f_p on Ω_X to ℝ such that the distribution of U = (f₁(X¹), …, f_p(X^p))^⊤ depends only on u^⊤Σu. Furthermore, X is said to follow a transelliptical graphical model with respect to a graph 𝓖 = (Γ, E) iff X has a transelliptical distribution and E = {(i, j) : (Σ⁻¹)_ij ≠ 0}. In this case we write X ~ TEG(𝓖). The following corollary states that ASG reduces to the transelliptical graphical model if the additive families 𝓐_Xⁱ are chosen to be span (f_i) and the underlying distribution of X is assumed to be transelliptical. Its proof is parallel to that of Theorem 3 and is omitted.

Corollary 1

Suppose

1. X has a transelliptical distribution with respect to a positive definite shape parameter Σ and injections f₁, …, f_p;

2. 𝓐_Xⁱ = span(f_i).

Then X ~ ASG(𝓖) iff X ~ TEG(𝓖).

4 Additive conditional covariance operator

In this and the next sections we introduce two linear operators that will give rise to two estimation procedures for ASG. The first is the additive conditional covariance operator (ACCO). Consider the bilinear form

$${\mathcal{A}}_U\times {\mathcal{A}}_V\to ℝ,(f,g)\mapsto E[f(U)g(V)].$$

By the Riesz representation theorem, this induces a bounded linear operator Σ_UV : 𝓐_V → 𝓐_U such that 〈f, Σ_UVg〈 = E[f (U)g(V)]. We call the operator

$${\mathrm{\sum }}_{UV\mid W}={\mathrm{\sum }}_{UV}-{\mathrm{\sum }}_{UW}{\mathrm{\sum }}_{WV}$$ (6)

the additive conditional covariance operator from V to U given W. This operator is different from the conditional variance operator introduced by Fukumizu, Bach, and Jordan (2009), because of the additive nature of the functional spaces 𝓐_U, 𝓐_V, and 𝓐_W. In particular, the following identity no longer holds:

$$\langle f,{\mathrm{\sum }}_{UV\mid W}g\rangle =E[cov(f(U),g(V)\mid W)].$$

In the following, for a Hilbert space 𝓗, a subspace 𝓛 of 𝓗, and an operator T defined on 𝓗, we use T|𝓛 to denote the operator T restricted on 𝓛. That is, T|𝓛 is a mapping from 𝓛 to 𝓗 with (T|𝓛)f = Tf for all f ∈ 𝓛. The following lemma gives a relation between the projection operator and the covariance operator.

Lemma 1

Let P_𝓐W: 𝓐_X → 𝓐_W be the projection on to 𝓐_W. Then P_𝓐W|𝓐_U = Σ_WU.

The next theorem shows that Σ_UV|W = 0 iff U ⫫_AV|W, which is the basis for our first ASG estimator. This resembles the relation between conditional covariance operator and conditional independence; that is, if C_UV|W is the conditional covariance operator in Fukumizu, Jordan, and Bach (2009), then C_UV|W = 0 iff U ⫫ V|W.

Theorem 5

U⫫_AV|W iff Σ_UV|W = 0.

5 Additive precision operator

We now introduce the second linear operator – the additive precision operator (APO), and show that a statement resembling the equivalence of (1) and (2) can be made for ASG if we replace the precision matrix by APO. As a result, the estimation of the ASG model can be turned into the sparse estimation of APO.

As a preparation, we first develop the notion of a matrix of operators on a direct-sum functional space. Let 𝓐_⊕X be the Hilbert space of functions consisting of the same set of functions as 𝓐_X but with the inner product

$${\langle f_1+\cdots +f_p,g_1+\cdots +g_p\rangle }_{{\mathcal{A}}_{\oplus X}}={\sum }_{i=1}^p{\langle f_i,g_i\rangle }_{{\mathcal{A}}_{X^i}}.$$

Suppose for each i, j we are given a linear operator T_X^jXⁱ: 𝓐_Xⁱ → 𝓐_X^j. Then we can construct a linear operator T : 𝓐_⊕X → 𝓐_⊕X as follows

$$T(f_1+\cdots +f_p)={\sum }_{i=1}^p{\sum }_{j=1}^p{T}_{X^j{X}^i}{f}_i.$$ (7)

This construction is legitimate because in the additive case f₁ + ··· + f_p uniquely determines (f₁, …, f_p). In fact, if we fix (c₂, …, c_p) in Ω_X² × ··· × Ω_X^p and vary x¹ in Ω_X¹, then using E[f₁(X¹)] = 0 we have

$$f_1(x^1)=f_1(x^1)+f_2{(c)}^2+\cdots +f_p(c^p)-E[f_1(X^1)+f_2(c^2)+\cdots +f_p(c^p)].$$

Conversely, a linear operator T : 𝓐_⊕X → 𝓐_⊕X uniquely determines operators T_X^jXⁱ : 𝓐_Xⁱ → 𝓐_X^j that satisfies (7), as follows. For each f_i ∈ 𝓐_⊕Xi, Tf_i can be represented as h₁ + ··· + h_p, where h_j ∈ 𝓐_X^j. Moreover, h₁ + ··· + h_p uniquely determines (h₁, …, h_p), and hence also h_j. We define T_X^jXⁱ : 𝓐_Xⁱ → 𝓐_X^j by T_X^jXⁱ(f_i) = h_j. Thus, there is a one-to-one correspondence between T and {T_X^jXⁱ} and they are related by (7). Henceforth we denote this one-to-one correspondence by the equality

$$T=\{T_{X^j{X}^i}:i,j=1,\dots ,p\}.$$

Moreover, T is bounded iff each T_X^jXⁱ is bounded, as shown in the next proposition.

Proposition 2

The following statements are equivalent

1. T_X^jXⁱ ∈ 𝓑 (𝓐_Xⁱ, 𝓐_X^j) for i, j = 1, …, p;

2. T ∈ 𝓑 (𝓐_⊕X, 𝓐_⊕X).

Our construction of T = {T_X^jXⁱ} echoes the construction of Bach (2008). However, our construction is relative to a different inner product and will play a new role under ACI. We are now ready to introduce the additive precision operator.

Definition 4

The operator ϒ : 𝓐_⊕X → 𝓐_⊕X defined by the matrix of operators {Σ_XⁱX^j: i ∈ Γ, j ∈ Γ} is called the additive covariance operator. If it is invertible then its inverse Θ is called the additive precision operator.

It can be easily checked that, for any f, g ∈ 𝓐_⊕X,

$${\langle f,\mathrm{\Upsilon }g\rangle }_{{\mathcal{A}}_{\oplus X}}=cov[f(X),g(X)]=\langle f,g\rangle .$$ (8)

This relation also shows why we used 𝓐_⊕X instead of 𝓐_X to define ϒ in the first place: if we had used 𝓐_X, then ϒ would have satisfied ϒf = f for any f ∈ 𝓐_X; that is, ϒ would have been the identity mapping from 𝓐_X to 𝓐_X. In the meantime, we note that, by construction, ϒ_XⁱX^j = Σ_XⁱX^j for all i, j = 1, …, p.

We should note that it is generally unreasonable to assume that the inverse of a covariance operator to be a bounded operator, because we typically assume the sequence of eigenvalues of such operators tend to 0. However, in our construction, the inner product 〈f_i, f_i〉_𝓐⊕X is the marginal variance var[f_i(Xⁱ)]. Thus our covariance operator ϒ is more like a correlation than a covariance. In particular, ϒ_ii are identity operators by construction. If we assume ϒ_ij are compact for all i ≠ j, then ϒ is of the form “identity plus compact perturbation”. Such operators, if invertible, are bounded from below by cI for some c > 0 (Bach, 2008), and therefore have bounded inverses. We summarize this fact in the following proposition.

Proposition 3

Suppose ϒ is invertible and, for each i ≠ j, ϒ_ij is compact. Then ϒ⁻¹ is bounded.

To establish the relation between the precision operator Θ and ACI, we need the block inversion formulas for operators. For any subvector U of X, let V denote U^c, and define the matrices of operators

$$\begin{array}{l}{T}_{UU}=\{T_{X^i{X}^j}:X_i\in U,X_j\in U\},T_{UV}=\{T_{X^i{X}^j}:X_i\in U,X_j\in V\},\\ T_{VU}=\{T_{X^i{X}^j}:X\in i\in V,X_j\in U\},T_{VV}=\{T_{X^i{X}^j}:X_i\in V,X_j\in V\}.\end{array}$$

From these definitions it follows that T_UU 𝓐_⊕U ⊆ 𝓐_⊕U, and

$${\langle f,T_{UU}g\rangle }_{{\mathcal{A}}_{\oplus X}}={\langle f,Tg\rangle }_{{\mathcal{A}}_{\oplus X}}\text{for}\text{any}f,g\in {\mathcal{A}}_{\oplus U}.$$

The same can be said of T_VV, T_UV, and T_VU. The following operator identities resemble the matrix identities derived from block-matrix inverse.

Lemma 2

Suppose that T ∈ 𝓑 (𝓐_⊕X, 𝓐_⊕X) is a self adjoint, positive definite operator. Then: the operators

$$T_{UU},T_{VV},T_{UU}-T_{UV}{T}_{VV}^{-1}{T}_{VU},T_{VV}-T_{VU}{T}_{UU}^{-1}{T}_{UV}$$

are bounded, self adjoint, and positive definite, and the following identities hold

$$\begin{array}{l}{(T^{-1})}_{UU}={(T_{UU}-T_{UV}{T}_{VV}^{-1}{T}_{VU})}^{-1},\\ {(T^{-1})}_{VV}={(T_{VV}-T_{VU}{T}_{UU}^{-1}{T}_{UV})}^{-1},\\ {(T^{-1})}_{UV}=-{(T^{-1})}_{UU}{T}_{UV}{T}_{VV}^{-1}=-T_{UU}^{-1}{T}_{UV}{(T^{-1})}_{VV},\\ {(T^{-1})}_{VU}=-{(T^{-1})}_{VV}{T}_{VU}{T}_{UU}^{-1}=-T_{VV}^{-1}{T}_{VU}{(T^{-1})}_{UU},\end{array}$$ (9)

where (T⁻¹)_UU and (T⁻¹)_VV are the blocks of T⁻¹ corresponding to 𝓐_⊕U and 𝓐_⊕V.

The above lemma shows that if ϒ has a bounded inverse, then Θ_UV is a well defined operator in 𝓑 (𝓐_⊕V, 𝓐_⊕U). Moreover, it behaves just like the off-diagonal block of the inverse of a covariance matrix. As we will see in the Appendix, manipulating the blocks of ϒ and Θ in the same way we do with a gaussian covariance matrix lead to the following equivalence.

Theorem 6

Suppose (U, V, W) is a partition of X and ϒ is invertible. Then U ⫫_A V|W iff Θ_UV = 0.

This relation is parallel to the equivalence between (1) and (2) in the gaussian case, which means we can turn estimating ASG into estimating the locations of the zero entries of APO, as shown in the next corollary.

Corollary 2

A random vector X follows an ASG(𝓖) model if and only if Θ_XⁱX^j = 0 whenever (i, j) ∉ E.

Note that the graph ASG(𝓖) is based entirely on pairwise additive conditional independence (4), which is similar to the pairwise Markov property for classical graphical models based on conditional independence. In the classical setting, pairwise Markov property with respect to a graph 𝓖 = (Γ, E) means

$$X^i⫫X^j\mid X^{-(i,j)}\iff (i,j)\notin E.$$

Under fairly mild conditions, this property also implies the global Markov property, which means if A, B, C are subsets of Γ and C separates A and B in 𝓖, then

$$X^A⫫X^B\mid X^C.$$

See Lauritzen (1996, Chapter 3). It is of theoretical interest to see if the parallel implication holds true in our additive setting. For convenience, let us agree to call property (4) the pairwise additive Markov property with respect to 𝓖. We are interested in whether X^A ⫫_AX^B|X^C holds whenever C separates A and B according to the graph 𝓖 determined by (4), a property that can be appropriately defined as the global additive Markov property with respect to 𝓖.

Using Theorem 6 we can easily derive a partial answer to this question, as follows. Suppose {A, B, C} is a partition of Γ and C separates A and B according to 𝓖 defined by (4). Then all paths between A and B intersects C. In other words, the block Θ_AB and Θ_BA of Θ consists of zero operators. However, by Theorem 6, this is equivalent to X^A ⫫_AX^B|X^C. We will leave it for future research to establish whether this statement is true when {A, B, C} is not necessarily a partition.

In the next two sections we introduce two estimators of the ASG model, one based on the ACCO introduced in the last section, and one based on the APO.

6 The ACCO-based estimator

By Theorem 5, it is reasonable to determine the absence of edges by smallness of the sample estimate of ||Σ_{XⁱX^j|X^−(i,j)}||. To develop this estimate, we first derive the finite-sample matrix representation of the operator Σ_{XⁱX^j|X^−(i,j)}.

For each i = 1, …, p, let κ_i : Ω_Xⁱ × ΩX^_i → ℝ be a positive definite kernel. Let

$${\mathcal{A}}_{X^i}=\text{span}\{{\kappa }_i(·,X_1^i)-E_n{\kappa }_i(X^i,X_1^i),\dots ,{\kappa }_i(·,X_n^i)-E_n{\kappa }_i(X^i,X_n^i)\},$$

where $E_n{\kappa }_i(X^i,X_k^i)=n^{-1}{\sum }_{\ell =1}^n{\kappa }_i(X_{\ell }^i,X_k^i)$. Let I_n be the n × n identity matrix and 1_n be the n-dimensional vector with all its entries equal to 1. Let Q be the projection matrix $I_n-1_n{1}_n^{\top }/n$. Let κ˚_i denote the vector-valued function

$$x^i\mapsto {({\kappa }_i(x^i,X_1^i)-E_n{\kappa }_i(X^i,X_1^i),\dots ,{\kappa }_i(x^i,X_n^i)-E_n{\kappa }_i(X^i,X_n^i))}^{\top }.$$

Then, any f ∈ 𝓐_Xⁱ can be expressed as $f={\stackrel{\circ }{\kappa }}_i^{\top }[f]$ for some [f] ∈ ℝⁿ, which is called the coordinate of f in the spanning system κ˚_i. A coordinate system is relative to a class of functions, and sometimes we need to use symbols such as [f]_𝓐Xⁱ for clarity. But in most of the discussions below this is unnecessary. In this coordinate notation we have ${(f(X_1^i),\dots ,f(X_n^i))}^{\top }={QK}_i[f]$, where $K_i=\{{\kappa }_i(X_k^i,X_{\ell }^i):k,\ell =1,\dots ,n\}$. Also, for any f, g ∈ 𝓐_Xⁱ,

$${\langle f,g\rangle }_{{\mathcal{A}}_{X^i}}=n^{-1}{({QK}_i[f])}^{\top }({QK}_i[g])=n^{-1}{[f]}^{\top }{K}_i{QK}_i[g].$$

Let K_−(i,j) denote the n(p − 2) × n matrix obtained by removing the ith and jth block of the np × n matrix (K₁, …, K_p)^⊤. Then, for any f ∈ 𝓐_X^−(i,j we have

$${(f(X_1^{-(i,j)}),\dots ,f(X_n^{-(i,j)}))}^{\top }={QK}_{-(i,j)}^{\top }[f]$$ (10)

for some [f] ∈ ℝ^n(p−2). Finally, for any linear operator T defined on a finite dimensional Hilbert space 𝓗, there is a matrix [T] such that [T f] = [T][f] for each function f ∈ 𝓗. The matrix [T] is called the matrix representation of T. This notation system is adopted from Horn and Johnson (1985) and was used in Li, Artemiou, and Li (2011), Li, Chun, and Zhao (2012), and Lee, Li, and Chiaromonte (2013).

Lemma 3

The following relations hold:

$$\begin{array}{c}{K}_i{QK}_i[{\mathrm{\sum }}_{X^i{X}^j}]=K_i{QK}_j,K_i{QK}_i[{\mathrm{\sum }}_{X^i{X}^{-(i,j)}}]=K_i{QK}_{-(i,j)}^{\top }\\ K_{-(i,j)}{QK}_{-(i,j)}^{\top }[{\mathrm{\sum }}_{X^{-(i,j)}{X}^i}]=K_{-(i,j)}{QK}_i.\end{array}$$ (11)

Because K_iQK_i and $K_{-(i,j)}{QK}_{-(i,j)}^{\top }$ are singular, we solve the equations in (11) by Tychonoff regularization:

$$\begin{array}{l}[{\mathrm{\sum }}_{X^i{X}^j}]={(K_i{QK}_i+{\epsilon }_1{I}_n)}^{-1}{K}_i{QK}_j,\\ [{\mathrm{\sum }}_{X^i{X}^{-(i,j)}}]={(K_i{QK}_i+{\epsilon }_1{I}_n)}^{-1}{K}_i{QK}_{-(i,j)}^{\top },\\ [{\mathrm{\sum }}_{X^{-(i,j)}{X}^i}]={(K_{-(i,j)}{QK}_{-(i,j)}^{\top }+{\epsilon }_2{I}_{n(p-2)})}^{-1}{K}_{-(i,j)}{QK}_i.\end{array}$$ (12)

Note that we use two different tuning constants ε₁ and ε₂, because the matrices involved have different dimensions. We will discuss how to choose these constants in Section 8. is then The matrix representation of Σ_{X^jXⁱ|X^−(i,j)} is then

$$[{\mathrm{\sum }}_{X^j{X}^i\mid X^{-(i,j)}}]=[{\mathrm{\sum }}_{X^j{X}^i}]-[{\mathrm{\sum }}_{X^j{X}^{-(i,j)}}][{\mathrm{\sum }}_{X^{-(i,j)}{X}^i}].$$ (13)

By definition, ||Σ_{X^jXⁱ|X^−(i,j)}||² is the maximum of

$${\langle {\mathrm{\sum }}_{X^j{X}^i\mid X^{-(i,j)}}f,{\mathrm{\sum }}_{X^j{X}^i\mid X^{-(i,j)}}f\rangle }_{{\mathcal{A}}_{X^j}}=n^{-1}{[f]}^{\top }{[{\mathrm{\sum }}_{X^j{X}^i\mid X^{-(i,j)}}]}^{\top }{K}_j{QK}_j[{\mathrm{\sum }}_{X^j{X}^i\mid X^{-(i,j)}}][f]$$

subject to 〈f, f〉_𝓐Xⁱ = [f]^⊤ K_iQK_i[f] = 1. To solve this generalized eigenvalue problem, let v = (K_iQK_i)^1/2[f]. Then, using inverse Tychonoff regularization, [f] = (K_iQK_i + ε₁I_n)^−1/2v. So it is equivalent to maximizing

$$v^{\top }{(K_i{QK}_i+{\epsilon }_1{I}_n)}^{-1/2}{[{\mathrm{\sum }}_{X^j{X}^i\mid X^{-(i,j)}}]}^{\top }{K}_j{QK}_j[{\mathrm{\sum }}_{X^j{X}^i\mid X^{-(i,j)}}]{(K_i{QK}_i+{\epsilon }_1{I}_n)}^{-1/2}v$$

subject to v^⊤ v = 1, which implies that ||Σ_{XⁱX^j|X^−(i,j)}||² is the largest eigenvalue of

$${\mathrm{\Lambda }}_{X^i{X}^j}={(K_i{QK}_i+{\epsilon }_1{I}_n)}^{-1/2}{[{\mathrm{\sum }}_{X^j{X}^i\mid X^{-(i,j)}}]}^{\top }{K}_j{QK}_j[{\mathrm{\sum }}_{X^j{X}^i\mid X^{-(i,j)}}]{(K_i{QK}_i+{\epsilon }_1{I}_n)}^{-1/2}.$$

Since the factor K_jQK_j[Σ_{X^jXⁱ|X^−(i,j)}] in the right-hand side is of the form

$$K_j{QK}_j{(K_j{QK}_j+{\epsilon }_1{I}_n)}^{-1}{K}_{j}Q(I_n-K_{-(i,j)}^{\top }{(K_{-(i,j)}{QK}_{-(i,j)}^{\top }+{\epsilon }_2{I}_{n(p-2)})}^{-1}{K}_{-(i,j)}){QK}_i,$$

it is reasonable to replace the first factor K_jQK_j by K_jQK_j + ε₁I_n, so that Λ_XⁱX^j becomes the quadratic form ${\mathrm{\Delta }}_{X^i{X}^j}{\mathrm{\Delta }}_{X^i{X}^j}^{\top }$, where

$$\begin{array}{l}{\mathrm{\Delta }}_{X^i{X}^j}={(K_i{QK}_i+{\epsilon }_1{I}_n)}^{-1/2}{K}_i\\ (Q-{QK}_{-(i,j)}^{\top }{(K_{-(i,j)}{QK}_{-(i,j)}^{\top }+{\epsilon }_2{I}_{n(p-2)})}^{-1}{K}_{-(i,j)}Q)K_j{(K_j{QK}_j+{\epsilon }_1{I}_n)}^{-1/2}.\end{array}$$ (14)

Hence, at the sample level, ||Σ_{XⁱX^j|X^−(i,j)}|| is the largest singular value of Δ_XⁱX^j. Because ${\mathrm{\Delta }}_{X^i{X}^j}^{\top }={\mathrm{\Delta }}_{X^j{X}^i}$, we have the symmetry ||Σ_{XⁱX^j|X^−(i,j)}|| = ||Σ_{X^jXⁱ|X^−(i,j)}||.

The above procedure involves inversion of $K_{-(i,j)}{QK}_{-(i,j)}^{\top }+{\epsilon }_2{I}_{n(p-2)}$, which is a large matrix if the graph 𝓖 contains many nodes. Nevertheless, as the next proposition shows the actual amount computation is not large — it is essentially that of spectral decomposition of an n by n matrix. Let diag(A₁, …, A_p) be the block-diagonal matrix with diagonal blocks A₁, …, A_p, and V be a matrix in ℝ^s×t, with s > t, which has singular value decomposition

$$V=(L_1,L_0)\text{diag}(D,0){(R_1,R_0)}^{\top },$$ (15)

where L₁ ∈ ℝ^s×u, L₀ ∈ ℝ^s×(s−u), R₁ ∈ ℝ^t×u, R₀ ∈ ℝ^t×(t−u), and D ∈ ℝ^u×u is a diagonal matrix with diagonal elements δ₁ ≥ ··· ≥ δ_u > 0.

Proposition 4

If the conditions in the last paragraph are satisfied, then

1. ${({VV}^{\top }+\epsilon I_s)}^{-1}={VR}_1{D}^{-1}({(D^2+\epsilon I_u)}^{-1}-{\epsilon }^{-1}{I}_u)D^{-1}{R}_1^{\top }{V}^{\top }+{\epsilon }^{-1}{I}_s$;

2. $V^{\top }{({VV}^{\top }+\epsilon I_s)}^{-1}V=R_{1}D{(D^2+\epsilon I_u)}^{-1}{DR}_1^{\top }$.

The above proposition allows us to calculate the inversions of s × s matrix (VV^⊤ + εI_s) essentially by spectral decomposition of t × t matrix V^⊤ V. We now summarize the estimation procedure as an algorithm. Suppose we observe an i.i.d. sample X₁, …, X_n from X, where X_i is a p-dimensional random vector ${(X_i^1,\dots ,X_i^p)}^{\top }$.

1. Standardize each component of X marginally. That is, for each i = 1, …, p, we standardize $X_1^i,\dots ,X_n^i$, so that E_nXⁱ = 0 and var_nXⁱ = 1.

2. Choose a positive definite kernel κ: ℝ × ℝ → ℝ, such as the gaussian radial kernel. Choose the kernel parameter γ using, for example, the procedure in Li, Chun, and Zhao (2012). Compute the kernel matrices K_i and K_−(i,j).

3. Compute Δ_XⁱX^j in (14), using part 2 of Proposition 4 to compute the matrix ${QK}_{-(i,j)}^{\top }{(K_{-(i,j)}{QK}_{-(i,j)}^{\top }+{\epsilon }_2{I}_{n(p-2)})}^{-1}{K}_{-(i,j)}Q$, with V = K_−(i,j)Q, s = n(p − 2), t = n, and u = n − 1. The tuning constants ε₁ and ε₂ are determined using the method described in Section 8.

4. For each i > j, compute the largest singular value of Δ_XⁱX^j. If it is smaller than a certain threshold, declare that there is no edge between i and j.

We emphasize again that throughout this procedure, the largest matrix whose spectral decomposition is needed is of dimension n × n, regardless of the dimension p of X.

7 The APO-based estimator

Our second estimator is based on Corollary 2, which states that Xⁱ ⫫ _AX^j|X^−(i,j) iff Θ_X^jXⁱ = 0. Thus we can determine the absence of edge between i, j if the sample version of ||Θ_X^jXⁱ|| is below a threshold. We first derive the matrix representations of the operator ϒ and its inverse Θ. Let K_1:p = (K₁, …, K_p)^⊤.

Theorem 7

The matrix representation of ϒ satisfies the following equation:

$$\text{diag}(K_1{QK}_1,\dots ,K_p{QK}_p)[\mathrm{\Upsilon }]=K_{1:p}{QK}_{1:p}^{\top }.$$ (16)

The matrix [Θ] is obtained by solving (16) for [ϒ]⁻¹ with Tychonoff regularization:

$$[\mathrm{\Theta }]={(K_{1:p}{QK}_{1:p}^{\top }+{\epsilon }_3{I}_{np})}^{-1}\text{diag}(K_1{QK}_1,\dots ,K_p{QK}_p).$$ (17)

To find ||Θ_X^jXⁱ||, we maximize

$${\langle {\mathrm{\Theta }}_{X^j{X}^i}{f}_i,{\mathrm{\Theta }}_{X^j{X}^i}{f}_i\rangle }_{{\mathcal{A}}_{X^j}}=n^{-1}{[f_i]}^{\top }{[{\mathrm{\Theta }}_{X^j{X}^i}]}^{\top }{K}_j{QK}_j[{\mathrm{\Theta }}_{X^j{X}^i}][f_i]$$

subject to 〈f_i, f_i〉_𝓐Xⁱ = [f_i]^⊤K_iQK_i[f_i] = 1. Following the similar development that leads to (14), we see that ||Θ_X^jXⁱ||² is the largest eigenvalue of

$${(K_i{QK}_i+{\epsilon }_1{I}_n)}^{-1/2}{[{\mathrm{\Theta }}_{X^j{X}^i}]}^{\top }(K_j{QK}_j+{\epsilon }_1{I}_n)[{\mathrm{\Theta }}_{X^j{X}^i}]{(K_i{QK}_i+{\epsilon }_1{I}_n)}^{-1/2},$$

where [Θ_X^jXⁱ] is the (j, i)th block of (17). Equivalently, ||Θ_X^jXⁱ|| is the largest singular value of

$${\mathrm{\Gamma }}_{X^i{X}^j}={(K_i{QK}_i+{\epsilon }_1{I}_n)}^{-1/2}{[{\mathrm{\Theta }}_{X^j{X}^i}]}^{\top }{(K_j{QK}_j+{\epsilon }_1{I}_n)}^{1/2}.$$

We now summarize the algorithm for the APO-based estimator. The first two steps are the same as in the ACCO-based algorithm; the last two steps are listed below.

• 3′ Compute [Θ] according to (17), where the inverse ${(K_{1:p}{QK}_{1:p}^{\top }+{\epsilon }_3{I}_{np})}^{-1}$ is computed using part 1 of Proposition 4, with V = K_1:pQ, ε = ε₃, s = np, and u = n − 1. The tuning constant ε₃ is determined by the method in Section 8.

• 4′ For each i, j = 1, …, p, i > j, compute the largest singular value of Γ_XⁱX^j. If it is smaller than a certain threshold, then there is no edge between i and j.

In concluding this section, we would like to mention that the operator norm used for both the ACCO and APO can be replaced by other norms, such as the Hilbert Schmidt norm. We have experimented both norms in our simulation studies (not presented), and found that their performances are similar.

8 Tuning parameters for regularization

In this section we propose cross-validation and generalized cross-validation procedures to select ε’s in the Tychonoff regularization. We need to invert three types of matrices

$$\text{type}\mathrm{I}:K_i{QK}_i,\text{type}\text{II}:K_{-(i,j)}{QK}_{-(i,j)}^{\top },\text{type}\text{III}:K_{1:p}{QK}_{1:p}^{\top }.$$

As before, we denote the constants in the regularized inverses as ε₁, ε₂, and ε₃.

Since our cross-validation is based on the total error in predicting one set of functions by another set of functions, we first formulate such predictions in generic terms. Let U and V be random vectors and W = (U^⊤, V^⊤)^⊤. Let 𝓕 = {f₀, f₁, …, f_r} be a set of functions of U, and 𝓖 = {g₀, g₁, …, g_s} be a set of functions of V, where g₀ and f₀ represent the constant function 1, which are included to account for any constant terms in prediction. Suppose we have an i.i.d. sample {W₁, …, W_n} of W. Let A = {a₁, …, a_m} ⊆ {1, …, n}, B = {b₁, …, b_n−m} = A^c. Let {W_a : a ∈ A} be the training set and {W_b : b ∈ B} be the testing set. Suppose ${\widehat{E}}_{V\mid U}^{(\epsilon )}:\mathcal{G}\to \mathcal{F}$ is a training-set-based operator such that, for each g ∈ 𝓖, ${\widehat{E}}_{V\mid U}^{(\epsilon )}g$ is the best prediction of g(V) based on functions in 𝓕, where ε is the tuning constant to be determined. The total error for predicting any function in 𝓖 evaluated at the testing set is

$${\sum }_{b\in B}{\sum }_{\mu =0}^s{[g_{\mu }(V_b)-({\widehat{E}}_{V\mid U}^{(\epsilon )}{g}_{\mu })(U_b)]}^2.$$ (18)

Now let L_U be the (r + 1) × m matrix whose ith row is the vector {f_i+1(U_a) : a ∈ A}, and L_V be the (s + 1) × m matrix whose kth row is the vector {g_k+1(V_a) : a ∈ A}. Let ℓ_U and ℓ_V denote the vector-valued functions (f₀, …, f_r)^⊤ and (g₀, …, g_r)^⊤. Then, following the argument similar to Lee, Li, and Chiaromonte (2013), we can express (18) as

$${\sum }_{b\in B}{\Vert {\ell }_V(V_b)-L_V{L}_U^{\top }{(L_U{L}_U^{\top }+\epsilon I_{r+1})}^{-1}{\ell }_U(U_b)\Vert }^2,$$

where ||·|| is the Euclidean norm. If we let G_U be the (r + 1) × (n − m) matrix whose bth column is ℓ_U (U_b), and G_V be the (s + 1) × (n − m) matrix whose bth column is ℓ_V (V_b), then the above can be further simplified as

$${\Vert G_V-L_V{L}_U^{\top }{(L_U{L}_U^{\top }+\epsilon I_{r+1})}^{-1}{G}_U\Vert }^2,$$

where ||·|| is the Frobenius matrix norm. Other matrix norms are also reasonable. In our simulation we used the operator norm, which performed well.

Return now to our cross-validation problem. In the k-fold cross validation, data are divided into k subsets. At each step, one subset is used as the testing set and union of the rest is used as the training set. Let i, j, … index the p components of x, a, b, … index the n subjects, and ν index the k steps in the k-fold cross-validation.

At the νth step, let {X_a : a ∈ A} be the training set and {X_b : b ∈ B} be the testing set. Reset κ˚_i to be the ℝ^m-valued function $x^i\mapsto {(\kappa (x^i,X_{a_1}^i),\dots ,\kappa (x^i,X_{a_m}^i))}^{\top }$. Let ℓ_i, ℓ_(i,j), and ℓ be the vector-valued functions defined by

$$\begin{array}{c}{\ell }_i(x^i)={(1,{\stackrel{\circ }{\kappa }}_i^{\top }(x^i))}^{\top },{\ell }_{(i,j)}(x^i,x^j)={(1,{\stackrel{\circ }{\kappa }}_i^{\top }(x^i),{\stackrel{\circ }{\kappa }}_j^{\top }(x^j))}^{\top },\\ \ell (x)={(1,{\stackrel{\circ }{\kappa }}_1^{\top }(x^1),\dots ,{\stackrel{\circ }{\kappa }}_p^{\top }(x^p))}^{\top }.\end{array}$$

Let ℓ_−i be the ℝ^n(p−1)+1-valued function obtained by removing κ˚_i from ℓ, and ℓ_−(i,j) be the ℝ^n(p−2)+1-valued function obtained by removing κ˚_i and κ˚_j from ℓ. Let L_i denote the matrix ( ${\ell }_i(X_{a_1}^i),\dots ,{\ell }_i(X_{a_m}^i)$), and define L_−i, L_(i,j), and L_−(i,j) similarly. Let G_i denote the matrix ( ${\ell }_i(X_{b_1}^i,\dots ,{\ell }_i(X_{b_{n-m}}^i)$), and define G_−i, G_(i,j), and G_−(i,j) similarly.

To determine ε₁, we use the functions in ℓ_i to predict the functions in ℓ_−i at X_b for any b ∈ B. That is,

$$\begin{array}{l}{\text{CV}}_1^{(\nu )}({\epsilon }_1)={\sum }_{i=1}^p{\sum }_{b\in B}{\Vert {\ell }_{-i}(X_b^{-i})-L_{-i}{L}_i^{\top }{(L_i{L}_i^{\top }+{\epsilon }_1{I}_{m+1})}^{-1}{\ell }_i(X_b^i)\Vert }^2\\ ={\sum }_{i=1}^p{\Vert G_{-i}-L_{-i}{L}_i^{\top }{(L_i{L}_i^{\top }+{\epsilon }_1{I}_{m+1})}^{-1}{G}_i\Vert }^2.\end{array}$$ (19)

Repeat this process for ν = 1, …, k and form ${\text{CV}}_1({\epsilon }_1)={\sum }_{\nu =1}^k{\text{CV}}_1^{(\nu )}({\epsilon }_1)$. We choose ε₁ by minimizing CV₁(ε₁) over a grid. To determine ε₂, we use the functions in ℓ_−(i,j) to predict the functions in ℓ_(i,j) at each X_b, b ∈ B, which leads to

$${\text{CV}}_2^{(\nu )}({\epsilon }_2)=2{\sum }_{i>j}{\Vert G_{(i,j)}-L_{(i,j)}{L}_{-(i,j)}^{\top }{(L_{-(i,j)}{L}_{-(i,j)}^{\top }+{\epsilon }_2{I}_{n(p-2)+1})}^{-1}{G}_{-(i,j)}\Vert }^2.$$ (20)

Let ${\text{CV}}_2({\epsilon }_2)={\sum }_{\nu =1}^k{\text{CV}}_2^{(\nu )}({\epsilon }_2)$. We choose ε₂ by minimizing CV₂(ε₂) over a grid. Here, we use part 1 of Proposition 4 to calculate ${(L_{-(i,j)}{L}_{-(i,j)}^{\top }+{\epsilon }_2{I}_{n(p-2)+1})}^{-1}$.

Alternatively, we can use the generalized cross validation criteria parallel to (19) and (20) to speed up computation (Wahba, 1990). Rather than splitting the data into training and testing samples, we compute the residual sum of squares from the entire data, and regularize it by the effective degrees of freedom as defined by the traces of relevant projection matrices. Specifically, the GCV-criterion parallel to (19) is

$${\text{GCV}}_1({\epsilon }_1)=\sum _{i=1}^p\frac{{\Vert L_{-i}-L_{-i}{L}_i^{\top }{(L_i{L}_i^{\top }+{\epsilon }_1{I}_{n+1})}^{-1}{L}_i\Vert }^2}{{\{1-\text{tr}(L_i^{\top }{(L_i{L}_i^{\top }+{\epsilon }_1{I}_{n+1})}^{-1}{L}_i)/n\}}^2}.$$

Here, L_i and L_−i are computed from the full data, and G_i in (19) is now replaced by L_i. Similarly, the GCV-criterion parallel to (20) is

$${\text{GCV}}_2({\epsilon }_2)=2\sum _{i>j}\frac{{\Vert L_{(i,j)}-L_{(i,j)}{L}_{-(i,j)}^{\top }{(L_{-(i,j)}{L}_{-(i,j)}^{\top }+{\epsilon }_2{I}_{n+1})}^{-1}{L}_{-(i,j)}\Vert }^2}{{\{1-\text{tr}(L_{-(i,j)}^{\top }{(L_{-(i,j)}{L}_{-(i,j)}^{\top }+{\epsilon }_2{I}_{n(p-2)+1})}^{-1}{L}_{-(i,j)})/n\}}^2}.$$

Here, again, the matrix $L_{-(i,j)}^{\top }{(L_{-(i,j)}{L}_{-(i,j)}^{\top }+{\epsilon }_2{I}_{n(p-2)+1})}^{-1}{L}_{-(i,j)}$ is to be calculated by part 2 of Proposition 4 with V = L_−(i,j), s = n(p − 2) + 1 and u = 1. Finally, we recommend taking ε₃ to be the same as ε₂. This is based on the intuition that $K_{-(i,j)}{QK}_{-(i,j)}^{\top }$ and $K_{1:p}{QK}_{1:p}^{\top }$ are of the similar forms and dimensions. This choice works well in our simulation studies.

From our experience, the GCV-criteria give the similar tuning constants as the CV-criteria but increase the computation speed by several folds. For this reason, we used the GCV criteria in our subsequent simulations and application.

9 Simulation studies

In this section we compare our ASG estimators with GGM and GCGM estimators, as proposed by Yuan and Lin (2007) and Liu et al (2013). Since our estimators are based on thresholding the norms of the operators Σ_{XⁱX^j|X^−(i,j)} and Θ_X^jXⁱ, it would be fair to compare them with the thresholded versions of GGM and GCGM. However, because L₁-regularization is widely used for these estimators, we will instead compare them with the L₁-regularized GGM and GCGM.

Comparison 1

We first compare the ASG estimators with GGM and GCGM when the copula assumption is violated. Consider the following models:

• Model I : X₁ = ε_1, $X_2=X_1^2+{\epsilon }_2$, X₃ = ε₃,

• Model II :X₁ = ε₁, X₂ = ε₂, $X_3=X_1^2+{\epsilon }_3,X_4=X_1^2+X_2^2+{\epsilon }_4$, X₅ = ε₅,

where ε₁, …, ε₅ are i.i.d. N(0, 1). The edge set for Model I is {(1, 2)}; the edge set for Model II is {(1, 3), (1, 4), (2, 4)}. The sample size is n = 100 and we repeated the simulation 100 times and the averaged ROC curves for correct estimation of the edge sets are presented in the upper panels of Figure 2. As expected, our ASG estimators perform much better than the GGM and GCGM when the copula assumption does not hold.

Figure 2.

Comparison of ACCO, APO, GGM, and GCGM. Upper panels: ROC curves for Models I and II, where the gaussian copula assumption is violated. Lower panels: ROC curves for Models III and IV, where the gaussian copula assumption holds.

For this and all the other comparisons, we have used the radial basis function (RBF) kernel with γ = 1/(2p), where γ is as it appears in the parametric form used in Li, Chun, and Zhao (2012). In Comparison 6 we give some results for alternative kernels and parameters.

Comparison 2

We now consider two models that satisfy the gaussian or copula gaus-sian assumption, which are therefore favorable to GGM and GCGM, to see how the ASG estimators compare with them:

• Model III : X ~ N(0, Θ⁻¹),

• Model IV : (f₁(X¹), …, f_p(X^p))^⊤ ~ N(0, Θ⁻¹),

where Θ is the 10 × 10 precision matrix with diagonal entries 1, 1, 1, 1.333, 3.010, 3.203, 1.543, 1.270, 1.554, and nonzero off-diagonal entries θ_3,5 = 1.418, θ_4,6 = −1.484, θ_4,10 = −0.744, θ_5,9 = 0.519, θ_5,10 = −0.577, θ_7,10 = 1.104, and θ_8,9 = −0.737. For Model IV, X¹, …, X^p each has a marginal gamma distribution with rate parameter 1 and shape parameter generated from unif(1, 5). The sample size is 50. The averaged ROC curves (over 100 replications) for the 4 estimators are shown in the lower panels in Figure 2. From these curves we can see the ASG estimators are comparable to GGM and GCGM when the gaussian or copula gaussian assumption hold.

The tuning parameters are selected by GCV₁ and GCV₂ from the coarse grids (5 × 10⁻⁴, …, 5 × 10³). The optimal tuning parameters do vary across repetitions, but the ballpark figures are relatively stable. For example, for model IV, the typical values are 5 × 10⁻⁴ for ε₁ and 5 × 10⁻¹ for ε₂. To evaluate the GCV criteria, in Figure 3 we compare the ROC curves using the GCV-tuned ε̂₁, ε̂₂ with with those for ε̂₁ × 10⁻¹ and ε̂₂ × 10⁻¹ and those for ε̂₁ × 10 and ε̂₂ × 10. We see that smaller tuning parameters performs better at low specificity, larger tuning parameters performs better at high specificity, but the GCV tuning criteria balances these two cases.

Figure 3.

Performance of GCV under Model IV.

Comparison 3

We now investigate the performance of our additive methods under a nonadditive model and compare with a fully nonparametric method that does not rely on the additive assumption. The fully nonparametric method is based on the conditional covariance operator (CCO) introduced by Fukumizu, Bach, and Jordan (2009). Consider the model

• Model V : X_i = ε_i, i = 1, 2, 5; X₃ = X₁ + ε₃; X₄ = (X₁ + X₂)² + ε₄,

where ε₁, …, ε₅ be i.i.d. N(0, 1). The nonadditivity is introduced through the term (X₁ + X₂)².

Let μ : ℝ^p−2 × ℝ^p−2 → ℝ be a positive definite kernel and M_X^−(i,j) be the kernel matrix { $\mu (X_r^{-(i,j)},X_s^{-(i,j)})$, s = 1, …, n}. Let

$$\begin{array}{c}{G}_{X^i}={QK}_{X^i}Q,G_{X^j}={QK}_{X^j}Q,G_{X^{-(i,j)}}={QM}_{X^{-(i,j)}}Q,\\ Q_{X^{-(i,j)}}({\epsilon }_2)=I_n-G_{X^{-(i,j)}}{(G_{X^{-(i,j)}}+{\epsilon }_2{I}_n)}^{-1}.\end{array}$$

Let V_{XⁱX^j|X−(^i,j)} be the CCO and V̂_{XⁱX^j|X^−(i,j)} be its sample estimate. Along the lines of Fukumizu, Bach, and Jordan (2009) we derive the operator norm V̂_{XⁱX^j|X^−(i,j)} as the largest singular value of

$${(G_{X^i}^{1/2}+{\epsilon }_1{I}_n)}^{1/2}{Q}_{X^{-(i,j)}}({\epsilon }_2){(G_{X^j}+{\epsilon }_1{I}_n)}^{1/2}.$$ (21)

We declare no edge between node i and j if this value is smaller than a threshold. A derivation of this is sketched in the Appendix. The tuning parameters are determined the similar GCV criteria as described in Section 8 except that the univariate kernel K_X^−(i,j) is replaced by the multivariate kernel M_X^−(i,j). We ran this simulation for n = 50 using ACCO, APO, GCGM, and this fully nonparametric estimator (denoted as fuku in the plot). The averaged ROC curves are shown in Figure 4. We can see that our methods work better Fukumizu et al’s method even in this nonadditive case.

Figure 4.

Comparison of ASG and the fully nonparametric estimator on Model V

Comparison 4

In this study we compare the 4 estimators by applying them to the data sets from the DREAM4 Challenges at http://wiki.c2b2.columbia.edu/dream/index.php?title=D4c2 in which subnetworks are extracted from a real biological networks, and gene expressions are simulated from stochastic differential equations, reflecting their biological interactions. Under this simulation scenario, two genes are not guaranteed to be linearly associated. In the DREAM 4 samples, p = 10, n = 31 for all five networks. In the analysis, we combined all data sets from various experiments (wildtype, knockout, knockdown and multifactorial) except for the time-course experiments, and apply each method to the combined data sets across all five network configurations in the DREAM4 Challenges. Figure 5 shows the averaged ROC curves for the 5 subnetworks. We note that APO and ACCO perform the best for the first subnetwork, and APO performs the best in the fifth subnetwork. In the other three subnetworks there is no clear separation among the four estimators. These comparisons suggest a possible benefit of our ASG estimators for finding non-linear biological interactions.

Figure 5.

Comparison of the ROC curves by ACCO, APO, GGM, GCGM estimators on 5 subnetworks in DREAM4 Challenges.

Comparison 5

In this study we increase the sample size n and dimension p to investigate the accuracy and computational feasibility of our methods. We use Model IV in Comparison 2 with (p, n) = (200, 400) and (p, n) = (400, 400). We compare APO with the fully nonparametric method and the averaged ROC curves are given in Figure 6. For larger p and n the ACCO is more computationally expensive than APO because one has to compute ||Σ̂_{XⁱX^j|X^−(i,j)}|| for all i, j = 1, …, p, i ≠ j. However, this problem is mitigated by APO, where the large (np × np) matrix [Θ] only needs to be calculated once.

Figure 6.

Comparison of ASG and fully nonparametric method for p = 200, n = 400 (left panel) and p = 400, n = 400 (right panel) using model IV

For p = 200 and n = 400, we selected (by GCV) ε̂₁ = 5 × 10⁻⁵, ε̂₂ = 0.5. Each run with fixed ε₁ and ε₂ took around 900 seconds (15 mins) in an Intel(R) Core(TM) i7-3770 CPU at 3.40GHz machine. For p = 400 and n = 400, we selected by GCV ε̂₁ = 5e⁻⁵, ε̂₂ = 0.5. Each run with fixed ε₁ and ε₂ took around 1900 seconds in the same computer. The GGM were not used for this simulation, since it took a very long time to get the estimates (more than a day for a single tuning parameter). We also performed 10 replications. It appears that our additive operator performs better than Fukumizu et al’s operator in high-dimensional cases.

Comparison 6

We now investigate the behavior of different kernels using Models III and IV. We have experimented with 3 types of kernels: the radial basis function kernel, the polynomial (POLY) kernel, and the rational quadratic (RATIONAL) kernel. The specific parametric forms we used for these kernels are as appear in Li, Chun, and Zhao (2012). For each kernel, we used two tuning parameters: for RBF, we use γ = 1/9, 1/(9p); for POLY, we use r = 2, 3; for RATIONAL, c = 200, 400. The ROC curves for these two models are given in Figure 7. We see that, with the exception of polynomial kernel with degree 3, these performance of these kernels are somewhat similar, with RBF, γ = 1/(9p) most favorable. The cubic polynomial kernel does not perform very well because it tends to amplify outlying observations, making their results unstable.

Figure 7.

Comparing different kernels using Model III (left panel) and Model IV (right panel).

10 Pathway analysis

We applied the four estimators to a real data set from Dimas et al (2009), in which gene expressions were measured from 85 individuals participating in the GenCord project. The same data set was also used in Chun, Zhang, and Zhao (2012). The participants were of Western European origin, and cell lines were obtained from their umbilical cords. In this analysis, we focused on one cell type, the B cells (lymphoblastoid cell lines). As in Chun et al (2012), we mapped the probes to 17,945 autosomal RefSeq genes, and use the resulting gene level data in our analysis. We then extracted the pathway information from the KEGG database (http://www.genome.jp/kegg/), and used it for partitioning genes and for evaluating the estimators by taking the interactions in the database as true interactions.

Due to the limited sample sizes, we only used the pathways whose sizes (i.e. number of genes) are greater than 2 and less than 36. There are a total of 300 such pathways. After ignoring directionality of regulations in KEGG pathways, we computed the false positives and true positives of the estimates and computed the area under curves (AUC) of the ROCs. We then evaluated the significance of ROC curves based on the p-values of the Wilcoxon rank sum test. Figure 8 shows the AUCs of all networks by ACCO and APO. The left panel shows AUCs based on ACCO, with significant values highlighted by green circles. The right panel shows those based on APO, with significant values highlighted by blue circles. In each case significance means the p-value is less than 10⁻⁴. We see that ACCO yields a larger number of significant ROCs than GGM. To reconfirm this pattern we computed the number of significant ROC curves by ACCO and GGM for three different significance levels, as presented in Table 2, which again shows that the numbers of significant ROC curves by ACCO are consistently higher at all three significance levels.

Figure 8.

Comparison of number of significant ROC curves by the ACCO and by GGM estimators for DREAM4 networks with p-values < 10⁻⁴.

Table 2.

Numbers of significant pathways with varying significance levels

p-values	ACCO	APO	GCGM	GGM
< 10−3	17	21	14	10
< 10−4	10	9	5	5
< 10−5	8	7	2	2

To provide further insights into the comparison, we listed in Table 3, Panel I, all the pathways with significant ROCs by ACCO (p-value < 10⁻⁵) whose AUCs by ACCO are larger than those by GGM. There were a total of 6 such pathways. In Table 3, Panel II, we listed all the pathways with significant ROCs by GGM whose AUCs by GGM are larger than those by ACCO. There were a total of 2 such pathways.

Table 3.

Significant pathways whose AUCs by ACCO are larger than those by GGM (panel I) and whose AUCs by ACCO are smaller than those by GGM (panel II)

Panel	Pathwaya	p	AUC
			ACCO	APO	GCGM	GGM
I	1	35	0.63	0.59	0.52	0.53
	2	24	0.66	0.66	0.56	0.58
	3	25	0.61	0.58	0.54	0.54
	4	18	0.67	0.60	0.62	0.59
	5	18	0.69	0.63	0.62	0.62
	6	16	0.69	0.59	0.55	0.55
II	7	14	0.60	0.65	0.62	0.66
	8	17	0.73	0.71	0.64	0.79

^aLabels in this column indicate the following pathways. 1: Activation of Csk by cAMP-dependent Protein Kinase Inhibits Signaling through the T Cell Receptor; 2: Cell Cycle: G2 M Checkpoint; 3: Cyclins and Cell Cycle Regulation; 4: Lck and Fyn tyrosine kinases in initiation of TCR Activation; 5: Th1 Th2 Differentiation; 6: hsa00710; 7: Proteasome Complex pathway; 8: Human Peptide GPCRs.

Some pathways were selected by more than one methods. Specifically,

$$\begin{array}{l}\#(\text{ACCO}\cap \text{APO})=4,\#(\text{ACCO}\cap \text{GGM})=1,\#(\text{ACCO}\cap \text{GCGM})=1,\\ \#(\text{APO}\cap \text{GGM})=1,\#(\text{APO}\cap \text{GCGM})=1,\#(\text{GGM}\cap \text{GCGM})=1,\end{array}$$

where #(A ∩ B) means the number of pathways found by method A and method B. It appears that ACCO and APO yield most similar results. Furthermore, single pathway was found by all methods.

From the analysis of these 300 pathways we see that the ASG estimators perform better than GGM and GCGM in a majority of cases. A plausible interpretation is that intrinsically nonlinear interactions (i.e. those that cannot be described by the gaussian copula assumption) are prevalent among genetic networks, and using a more flexible interaction assumption such as the ASG has real benefits in modeling of gene regulations.

11 Conclusions

In this paper we introduced two new estimators for graphical models based on the notion of additive conditional independence. They substantially broaden the applicability of the current graphical model estimators based on the gaussian or copula gaussian assumption, but at the same time avoid using high-dimensional kernels that can hamper estimation accuracy. As we have demonstrated by simulation and data analysis, the new methods work particularly well when the interactions exhibit intrinsic nonlinearity, which cannot be captured by the gaussian or copula gaussian models, and are comparable to existing methods when the copula gaussian assumption holds.

Our generalization of precision matrix to additive precision operator allows us to adopt essentially the same estimation strategy of the gaussian graphical model at the operator level. Using this generalization we succeeded in preserving the simplicity of the gaussian dependence structure without suffering from its austere distributional restriction.

This basic idea opens up many other possibilities that cannot be explored within the scope of this paper. For example, although both the ACCO and APO estimators are based on truncating small operators (as measured by the operator norm), it is conceivable that more sophisticated sparse techniques, such as LASSO, SCAD, and adaptive LASSO, can be adapted to this setting as well. How to implement these techniques in the operator setting, and how much they would improve the truncation-based estimators presented here, would be interesting questions for future research. Moreover, the asymptotic properties of the proposed estimators, such as consistency and asymptotic distributions, also need to be developed in further studies.

Appendix: proofs

We first state without proof several simple geometric facts that will be useful for proving Theorems 1 and 2. Let P_𝓛 be the projection on to a subspace 𝓛 of L₂(P_X), U and W be subvectors of X. Then

$${\mathcal{A}}_{U\cup W}={\mathcal{A}}_U+{\mathcal{A}}_W.$$ (22)

$$P_{{\mathcal{A}}_{U\cup W}}=P_{{\mathcal{A}}_{U\cup W}\ominus {\mathcal{A}}_W}+P_{{\mathcal{A}}_W}.$$ (23)

$${\mathcal{A}}_{U\cup W}\ominus {\mathcal{A}}_W=(I-P_{{\mathcal{A}}_W}){\mathcal{A}}_U$$ (24)

Proof of Theorem 1

1 ⇔ 2 follows from relation (22); 2 ⇔ 3 follows from relation (24); 6 ⇔ 7 follows from (23); 4 ⇔ 5 follows from the relation (23).

2 ⇒ 4. By statement 2, P_{𝓐V∪W ⊖ 𝓐W} (𝓐_U∪W ⊖ 𝓐_W) = {0}. By relation (24) and the fact that 𝓐_V∪W ⊖ 𝓐_W ⊥ 𝓐_W, we have

$$P_{{\mathcal{A}}_{V\cup W}\ominus {\mathcal{A}}_W}({\mathcal{A}}_{U\cup W}\ominus {\mathcal{A}}_W)=P_{{\mathcal{A}}_{V\cup W}\ominus {\mathcal{A}}_W}(I-P_{{\mathcal{A}}_W}){\mathcal{A}}_U=P_{{\mathcal{A}}_{V\cup W}\ominus {\mathcal{A}}_W}{A}_U.$$

4 ⇒ 6. This is because

$$P_{{\mathcal{A}}_{V\cup W}\ominus {\mathcal{A}}_W}{\mathcal{A}}_{U\cup W}=P_{{\mathcal{A}}_{V\cup W}\ominus {\mathcal{A}}_W}({\mathcal{A}}_U+{\mathcal{A}}_W)=P_{{\mathcal{A}}_{V\cup W}\ominus {\mathcal{A}}_W}{\mathcal{A}}_W=\{0\},$$

where the first equality follows from (22); the second from assertion 4; the third from 𝓐_W ⊥ 𝓐_V∪W ⊖ 𝓐_W.

6 ⇒ 2. Because ker(P_{𝓐V∪W ⊖ 𝓐W}) = [ran(P_{𝓐V∪W ⊖ 𝓐W})]^⊥ = (𝓐_V∪W ⊖ 𝓐_W)^⊥, we have

$${\mathcal{A}}_{U\cup W}\perp {\mathcal{A}}_{V\cup W}\ominus {\mathcal{A}}_W,$$

which implies 𝓐_U∪W ⊖ 𝓐_W ⊥ 𝓐_V∪W ⊖ 𝓐_W because 𝓐_U∪W ⊖ 𝓐_W ⊆ 𝓐_U∪W.

Proof of Theorem 2

Axiom 1 is obvious. To prove 2, 3, 4, let U, V, W, R be disjoint sub-vectors of X and 𝓐_U, 𝓐_V, 𝓐_W, 𝓐_R be their respective additive families.

2. If U ⫫_A V ∪ R|W, then

$$({\mathcal{A}}_U+{\mathcal{A}}_W)\ominus {\mathcal{A}}_W\perp ({\mathcal{A}}_{V\cup R}+{\mathcal{A}}_W)\ominus {\mathcal{A}}_W.$$

Axiom 2 follows because (𝓐_V + 𝓐_W) ⊖ 𝓐_W ⊆ (𝓐_V∪R + 𝓐_W) ⊖ 𝓐_W.

3. By Theorem 1, part 3, U ⫫_A V ∪ R|W is equivalent to

$$(I-P_{{\mathcal{A}}_W}){\mathcal{A}}_U\perp (I-P_{{\mathcal{A}}_W}){\mathcal{A}}_{V\cup R}$$ (25)

By Axiom 2, we have U ⫫_A R|W, which, by Theorem 1, part 5, implies P_𝓐W f = P_𝓐W∪R f for all f ∈ 𝓐_U. Hence the left hand side of (25) can be rewritten as (I − P_𝓐W∪R) 𝓐_U. Meanwhile, the right hand side of (25) contains (I − P_𝓐W)𝓐_V. Hence

$$(I-P_{{\mathcal{A}}_{W\cup R}}){\mathcal{A}}_U\perp (I-P_{{\mathcal{A}}_W}){\mathcal{A}}_V.$$ (26)

Because 𝓐_W ⊆ 𝓐_W∪R, we have ran(I − P_𝓐W∪R) ⊥ ran(P_𝓐W∪R). Hence

$$(I-P_{{\mathcal{A}}_{W\cup R}}){\mathcal{A}}_U\perp P_{{\mathcal{A}}_{W\cup R}\ominus {\mathcal{A}}_W}{\mathcal{A}}_V.$$ (27)

Because

$$(I-P_{{\mathcal{A}}_{W\cup R}}){\mathcal{A}}_V=(I-P_{{\mathcal{A}}_W}-P_{{\mathcal{A}}_{W\cup R}\ominus {\mathcal{A}}_W}){\mathcal{A}}_V\subseteq (I-P_{{\mathcal{A}}_W}){\mathcal{A}}_V+P_{{\mathcal{A}}_{W\cup R}\ominus {\mathcal{A}}_W}{\mathcal{A}}_V,$$

relation (26) and the second expression in (27) implies the desired relation

$$(I-P_{{\mathcal{A}}_{W\cup R}}){\mathcal{A}}_U\perp (I-P_{{\mathcal{A}}_{W\cup R}}){\mathcal{A}}_V.$$

4. By Theorem 1, part 3, U ⫫_AV|W and U ⫫_AR|W ∪ V are equivalent to

$$(I-P_{{\mathcal{A}}_W}){\mathcal{A}}_U\perp (1-P_{{\mathcal{A}}_W}){\mathcal{A}}_V,(I-P_{{\mathcal{A}}_{V\cup W}}){\mathcal{A}}_U\perp (I-P_{{\mathcal{A}}_{V\cup W}}){\mathcal{A}}_R.$$ (28)

We need to show (I − P_𝓐W) 𝓐_U ⊥ (I − P_𝓐W)(𝓐_V + 𝓐_R), which, by the first relation in in (28), is implied by

$$(I-P_{{\mathcal{A}}_W}){\mathcal{A}}_U\perp (I-P_{{\mathcal{A}}_W}){\mathcal{A}}_R.$$ (29)

Because U ⫫_A V|W, the left side of (29) is (I − P_𝓐V∪W) 𝓐_U. By (23), the right side of (29) is (I − P_𝓐V∪W + P_{𝓐V∪W ⊖ 𝓐W}) 𝓐_R. So, under (28), relation (29) is equivalent to

$$(I-P_{{\mathcal{A}}_{V\cup W}}){\mathcal{A}}_U\perp (I-P_{{\mathcal{A}}_{V\cup W}}+P_{{\mathcal{A}}_{V\cup W}\ominus A_W}){\mathcal{A}}_R.$$

Since (I − P_𝓐V∪W) 𝓐_U ⊥ (I − P_𝓐V∪W) 𝓐_R, it suffices to show that

$$(I-P_{{\mathcal{A}}_{V\cup W}}){\mathcal{A}}_U\perp P_{{\mathcal{A}}_{V\cup W}\ominus {\mathcal{A}}_W}{\mathcal{A}}_R,$$ (30)

which is true because

$$\text{ran}(P_{{\mathcal{A}}_{V\cup W}\ominus {\mathcal{A}}_W})={\mathcal{A}}_{V\cup W}\ominus {\mathcal{A}}_W=(I-P_{{\mathcal{A}}_W})({\mathcal{A}}_{V\cup W})=\text{ran}((I-P_{{\mathcal{A}}_W})P_{{\mathcal{A}}_{V\cup W}})$$

and (I − P_𝓐V∪W)(I − P_𝓐W) P_𝓐V∪W = 0.

Proof of Theorem 3

The statement U ⫫_A V|W means

$$\langle f-P_{{\mathcal{A}}_W}f,g-P_{{\mathcal{A}}_W}g\rangle =0\forall f\in {\mathcal{A}}_U,g\in {\mathcal{A}}_V.$$ (31)

Let A, B, and C be the sets of indices for U, V, and W and h(W) be the vector {f_k(X^k) : k ∈ C}. By condition 3, f and g can be written as Σ_i∈A α_if_i and Σ_j∈B β_j f_j for some α = (α₁, …, α_r)^⊤ and β = (β₁, …, β_s)^⊤. Since (f(U), h(W)) is jointly gaussian, the conditional expectation E[f(U)|h(W)] belongs to 𝓐_W. For the same reason E[g(V)|h(W)] belongs to 𝓐_W. It follows that P_𝓐Wf = E[f(U)|h(W)], P_𝓐Wg = E[g(V)|h(W)], and

$$\langle f-P_{{\mathcal{A}}_W}f,g-P_{{\mathcal{A}}_W}g\rangle =E[cov(f(U),g(V)\mid h(W))]=cov[f(U),g(V)\mid h(W)],$$

where the second equality holds because the random vector (f(U), g(V), h(W)) is multivariate gaussian. Consequently, (31) is equivalent to f(U) ⫫ g(V)|h(W) for all f ∈ 𝓐_U and g ∈ 𝓐_V, which is equivalent to

$$\{f_i(X^i):i\in A\}⫫\{f_j(X^j):j\in B\}\mid h(W).$$

Because f₁, …, f_p are injective, this is equivalent to U ⫫ V|W.

Proof of Proposition 1

Let f ∈ L₂(P_U) and g ∈ L₂(P_V) be any functions with ||f|| = ||g|| = 1. Because P_𝓐W is idempotent and self adjoint, we have

$$\langle f-P_{{\mathcal{A}}_W}f,g-P_{{\mathcal{A}}_W}g\rangle =\langle f,g\rangle -\langle f,P_{{\mathcal{A}}_W}g\rangle .$$

For each γ = 1, …, p − 2, let { ${\psi }_{\alpha \gamma }^w$: α = 1, 2, …} be the hermite polynomials in the variable w^γ. Similarly, let { ${\psi }_{\alpha }^u$: α = 1, 2, …} and { ${\psi }_{\alpha }^v$: α = 1, 2, …} be the hermite polynomials in the variables u and v. By Lancaster (1957), these polynomials form orthonormal bases in L₂(P_W^γ), L₂(P_U), and L₂(P_V). It follows that the set { ${\psi }_{\alpha \gamma }^w$: α = 1, 2, …, γ = 1, …, p − 2} is an orthonormal basis for 𝓐_W. Moreover, by Lancaster (1957), whenever α ≠ β,

$$\{{\psi }_{\alpha }^u\}\cup \{{\psi }_{\alpha }^v\}\cup \{{\psi }_{\alpha \gamma }^w:\gamma =1,\dots ,p-2\}\perp \{{\psi }_{\beta }^u\}\cup \{{\psi }_{\beta }^v\}\cup \{{\psi }_{{\beta }_{\gamma }}^w:\gamma =1,\dots ,p-2\}.$$

So, if we let ${\mathcal{A}}_W^{\alpha }=\text{span}\{{\psi }_{\alpha \gamma }^w:\gamma =1,\dots ,p-2\}$, then $P_{{\mathcal{A}}_W}g={\sum }_{\alpha =1}^{\infty }{P}_{{\mathcal{A}}_W^{\alpha }}g$. Each term in this sum is

$$P_{{\mathcal{A}}_W^{\alpha }}g=P_{{\mathcal{A}}_W^{\alpha }}({\sum }_{\beta =1}^{\infty }\langle g,{\psi }_{\beta }^v\rangle {\psi }_{\beta }^v)={\sum }_{\beta =1}^{\infty }\langle g,{\psi }_{\beta }^v\rangle P_{{\mathcal{A}}_W^{\alpha }}{\psi }_{\beta }^v=\langle g,{\psi }_{\alpha }^v\rangle P_{{\mathcal{A}}_W^{\alpha }}{\psi }_{\alpha }^v$$

By Lancaster (1957) again,

$$\langle {\psi }_{\alpha }^v,{\psi }_{\alpha \gamma }^w\rangle ={\rho }_{V{W}^{\gamma }}^{\alpha },\langle {\psi }_{\alpha \gamma }^w,{\psi }_{\alpha \delta }^w\rangle ={\rho }_{W^{\gamma }{W}^{\delta }}^{\alpha },$$

which yield $P_{{\mathcal{A}}_W^{\alpha }}{\psi }_{\alpha }^v=R_{VW}^{\odot \alpha }{(R_{WW}^{\odot \alpha })}^{-1}{\psi }_{\alpha \ast }^w$, where ${\psi }_{\alpha \ast }^w={({\psi }_{\alpha 1}^w,\dots ,{\psi }_{\alpha ,p-2}^w)}^{\top }$. Hence

$$P_{{\mathcal{A}}_W}g={\sum }_{\alpha =1}^{\infty }{P}_{{\mathcal{A}}_W^{\alpha }}g={\sum }_{\alpha =1}^{\infty }\langle g,{\psi }_{\alpha }^v\rangle R_{VW}^{\odot \alpha }{(R_{WW}^{\odot \alpha })}^{-1}{\psi }_{\alpha \ast }^w.$$

Using this we compute

$$\begin{array}{l}\langle f,P_{{\mathcal{A}}_W}g\rangle ={\sum }_{\alpha =1}^{\infty }\langle g,{\psi }_{\alpha }^v\rangle \langle f,R_{VW}^{\odot \alpha }{(R_{WW}^{\odot \alpha })}^{-1}{\psi }_{\alpha \ast }^w\rangle \\ ={\sum }_{\alpha =1}^{\infty }\langle g,{\psi }_{\alpha }^v\rangle {\sum }_{\beta =1}^{\infty }\langle f,{\psi }_{\beta }^u\rangle \langle {\psi }_{\beta }^u,R_{VW}^{\odot \alpha }{(R_{WW}^{\odot \alpha })}^{-1}{\psi }_{\alpha \ast }^w\rangle \\ ={\sum }_{\alpha =1}^{\infty }\langle g,{\psi }_{\alpha }^v\rangle \langle f,{\psi }_{\alpha }^u\rangle \langle {\psi }_{\alpha }^u,R_{VW}^{\odot \alpha }{(R_{WW}^{\odot \alpha })}^{-1}{\psi }_{\alpha \ast }^w\rangle \\ ={\sum }_{\alpha =1}^{\infty }\langle g,{\psi }_{\alpha }^v\rangle \langle f,{\psi }_{\alpha }^u\rangle R_{VW}^{\odot \alpha }{(R_{WW}^{\odot \alpha })}^{-1}{R}_{WU}^{\odot \alpha }.\end{array}$$

By the similar calculation we found $\langle f,g\rangle ={\sum }_{\alpha =1}^{\infty }\langle f,{\psi }_{\alpha }^u\rangle \langle g,{\psi }_{\alpha }^v\rangle {\rho }_{UV}^{\alpha }$. Hence

$$\langle f,g\rangle -\langle f,P_{{\mathcal{A}}_W}g\rangle ={\sum }_{\alpha =1}^{\infty }\langle g,{\psi }_{\alpha }^v\rangle \langle f,{\psi }_{\alpha }^u\rangle ({\rho }_{UV}^{\alpha }-R_{VW}^{\odot \alpha }{(R_{WW}^{\odot \alpha })}^{-1}{R}_{WU}^{\odot \alpha })$$

Finally, by the Cauchy-Schwarz inequality,

$$\begin{array}{l}{(\langle f,g\rangle -\langle f,P_{{\mathcal{A}}_W}g\rangle )}^2\le {\sum }_{\alpha =1}^{\infty }{\langle g,{\psi }_{\alpha }^v\rangle }^2{\sum }_{\alpha =1}^{\infty }{\langle f,{\psi }_{\alpha }^u\rangle }^2{({\rho }_{UV}^{\alpha }-R_{VW}^{\odot \alpha }{(R_{WW}^{\odot \alpha })}^{-1}{R}_{WU}^{\odot \alpha })}^2\\ ={\sum }_{\alpha =1}^{\infty }{\langle f,{\psi }_{\alpha }^u\rangle }^2{({\rho }_{UV}^{\alpha }-R_{VW}^{\odot \alpha }{(R_{WW}^{\odot \alpha })}^{-1}{R}_{WU}^{\odot \alpha })}^2\\ \le max\{{({\rho }_{UV}^{\alpha }-R_{VW}^{\odot \alpha }{(R_{WW}^{\odot \alpha })}^{-1}{R}_{WU}^{\odot \alpha })}^2:\alpha =1,2,\dots \},\end{array}$$

as desired.

Proof of Lemma 1

For any f ∈ 𝓐_U and g ∈ 𝓐_W, we have

$$\langle (P_{{\mathcal{A}}_W}\mid {\mathcal{A}}_U-{\mathrm{\sum }}_{WU})f,g\rangle ={\langle P_{{\mathcal{A}}_W}f,g\rangle }_{{\mathcal{A}}_W}-\langle {\mathrm{\sum }}_{WU}f,g\rangle .$$ (32)

The first term on the right-hand side is

$$\langle P_{{\mathcal{A}}_W}f,g\rangle =\langle f,P_{{\mathcal{A}}_W}g\rangle =\langle f,g\rangle =E[f(U)g(W)].$$

Hence 〈(P_𝓐W ∣ 𝓐_U − Σ_WU)f,g〉_𝓐W = 0 for all f ∈ 𝓐_W and g ∈ 𝓐_U.

Proof of Theorem 5

If U ⫫_A V|W, then, for any f ∈ 𝓐_U, g ∈ 𝓐_V,

$$0=\langle (I-P_{{\mathcal{A}}_W})f,(I-P_{{\mathcal{A}}_W})g\rangle =\langle f,g\rangle -\langle f,P_{{\mathcal{A}}_W}g\rangle ,$$ (33)

where the second equality follows from the definition of a projection. Using Lemma 1, we calculate the two terms on the right as

$$\langle f,g\rangle =\langle f,{\mathrm{\sum }}_{UV}g\rangle ,\langle f,P_{{\mathcal{A}}_W}g\rangle =\langle f,{\mathrm{\sum }}_{WV}g\rangle =\langle f,{\mathrm{\sum }}_{UW}{\mathrm{\sum }}_{WV}g\rangle .$$

Hence (33) is equivalent to 〈f, (Σ_UV − Σ_UW Σ_WV)g〉 = 0, which implies Σ_UV − Σ_UW Σ_WV = 0. The arguments can obviously be reversed.

Proof of Proposition 2

1 ⇒ 2. For any f ∈ 𝓐_⊕X, ∣∣f∣∣_𝓐⊕X ≤ 1, we have, by the triangular inequality,

$$\begin{array}{l}{\Vert Tf\Vert }_{{\mathcal{A}}_{\oplus X}}\le {\sum }_{i=1}^p{\sum }_{j=1}^p\Vert T_{X^j{X}^i}{f}_i\Vert \\ \le {\sum }_{i=1}^p{\sum }_{j=1}^p\Vert T_{X^j{X}^i}\Vert \Vert f_i\Vert \le p^2{max}_{i,j=1,\dots ,p}\Vert T_{X^j{X}^i}\Vert <\infty .\end{array}$$

2 ⇒ 1. For any f_k ∈ 𝓐_X^k and ∣∣f_k∣∣_𝓐k ≤ 1, we have

$${\Vert T{f}_k\Vert }_{{\mathcal{A}}_{\oplus X}}^2={\langle T{f}_k,T{f}_k\rangle }_{{\mathcal{A}}_{\oplus X}}={\sum }_{i=1}^p\langle T_{X^i{X}^k}{f}_k,T_{X^i{X}^k}{f}_k\rangle \ge {\Vert T_{X^{\ell }{X}^k}{f}_k\Vert }^2,$$

which implies

$$\begin{array}{l}sup\{\Vert T_{X^{\ell }{X}^k}{f}_k\Vert :\Vert f_k\Vert \le 1\}\le sup\{{\Vert T{f}_k\Vert }_{{\mathcal{A}}_{\oplus X}}:\Vert f_k\Vert \le 1\}\\ =sup\{{\Vert T{f}_k\Vert }_{{\mathcal{A}}_{\oplus X}}:{\Vert f_k\Vert }_{{\mathcal{A}}_{\oplus X}}\le 1\}=\Vert T\Vert <\infty .\end{array}$$

Proof of Lemma 2

First, we show that T_UU (and hence also T_VV) is bounded, self adjoint, and positive definite. Let h ∈ 𝓐_⊕U. Then T_UU h ∈ 𝓐_⊕U. Hence

$${\langle T_{UU}h,T_{UU}h\rangle }_{{\mathcal{A}}_{\oplus X}}={\langle T_{UU}h,Th\rangle }_{{\mathcal{A}}_{\oplus X}}\le {\Vert T_{UU}h\Vert }_{{\mathcal{A}}_{\oplus X}}{\Vert Th\Vert }_{{\mathcal{A}}_{\oplus X}}$$

which implies that ∣∣T_UUh∣∣_𝓐⊕X is either 0 or no greater than ||T|| ||h||. Hence T_UU is bounded. That T_UU is self adjoint follows from 〈g, T_UUh〉_𝓐⊕X = 〈g, Th〉_𝓐⊕X for all g, h ∈ 𝓐_⊕U. Because T > 0, if h ∈ 𝓐_⊕U and h ≠ 0, then 〈h, Th〉_𝓐⊕X > 0. Because 〈h, Th〉_𝓐⊕X = 〈h, T_UUh〉_𝓐⊕X we have 〈h, T_UUh〉_𝓐⊕X > 0. Thus T_UU > 0.

That $T_{UU}-T_{UW}{T}_{WW}^{-1}{T}_{WU}$ and $T_{VV}-T_{VW}{T}_{WW}^{-1}{T}_{WV}$ are bounded and self adjoint can be proved similarly. To show that they are positive definite and the subsequent identities, let I_UU be the identity operator from 𝓐_⊕U to 𝓐_⊕U, and O_UV be the zero operator from 𝓐_⊕V to 𝓐_⊕U. Define I_VV and O_VU similarly. Then

$$\left(\begin{array}{cc}{(T^{-1})}_{UU}& {(T^{-1})}_{UV}\\ {(T^{-1})}_{VU}& {(T^{-1})}_{VV}\end{array}\right)\left(\begin{array}{cc}{T}_{UU}& T_{UV}\\ T_{VU}& T_{VV}\end{array}\right)=\left(\begin{array}{cc}{I}_{UU}& O_{UV}\\ O_{VU}& I_{VV}\end{array}\right).$$

Hence

$${(T^{-1})}_{UU}{T}_{UU}+{(T^{-1})}_{UV}{T}_{VU}=I_{UU}$$ (34)

$${(T^{-1})}_{UU}{T}_{UV}+{(T^{-1})}_{UV}{T}_{VV}=O_{UV}.$$ (35)

By part 1, T_{V V} is invertible. Hence, by (35),

$${(T^{-1})}_{UV}=-{(T^{-1})}_{UU}{T}_{UV}{T}_{VV}.$$ (36)

Substitute this into (34) to obtain

$${(T^{-1})}_{UU}(T_{UU}-T_{UV}{T}_{VV}{T}_{VU})=I_{UU}$$ (37)

which implies that T_UU − T_UV T_VV T_VU is invertible and the first two identities in (9) hole. Similarly, we can prove that T_VV − T_VU T_UU T_UV is invertible and the last two identities in (9) hold.

Proof of Theorem 6

Let A, B, C be the index sets of U, V, W, Z = U ∪ V, and

$$\begin{array}{l}{\mathrm{\Upsilon }}_{ZZ}=\{{\mathrm{\sum }}_{X^i{X}^j}:i,j\in A\cup B\},{\mathrm{\Upsilon }}_{ZW}=\{{\mathrm{\sum }}_{X^i{X}^j}:i\in A\cup B,j\in C\},\\ {\mathrm{\Upsilon }}_{WZ}=\{{\mathrm{\sum }}_{X^i{X}^j}:i\in C,j\in A\cup B\},{\mathrm{\Upsilon }}_{WW}=\{{\mathrm{\sum }}_{X^i{X}^j}:i\in C,j\in C\}.\end{array}$$

These operators are different from the operators such as Σ_UV in Section 4 because they are defined by different inner products. For example, Σ_UU is the identity mapping but ϒ_UU is not. By Lemma 2, ${\mathrm{\Theta }}_{ZZ}={({\mathrm{\Upsilon }}_{ZZ}-{\mathrm{\Upsilon }}_{ZW}{\mathrm{\Upsilon }}_{WW}^{-1}{\mathrm{\Upsilon }}_{WZ})}^{-1}$. Because

$${\mathrm{\Upsilon }}_{ZZ}-{\mathrm{\Upsilon }}_{ZW}{\mathrm{\Upsilon }}_{WW}^{-1}{\mathrm{\Upsilon }}_{WZ}=\left(\begin{array}{cc}{\mathrm{\Upsilon }}_{UU}-{\mathrm{\Upsilon }}_{UW}{\mathrm{\Upsilon }}_{WW}^{-1}{\mathrm{\Upsilon }}_{WU}& {\mathrm{\Upsilon }}_{UV}-{\mathrm{\Upsilon }}_{UW}{\mathrm{\Upsilon }}_{WW}^{-1}{\mathrm{\Upsilon }}_{WV}\\ {\mathrm{\Upsilon }}_{VU}-{\mathrm{\Upsilon }}_{VW}{\mathrm{\Upsilon }}_{WW}^{-1}{\mathrm{\Upsilon }}_{WU}& {\mathrm{\Upsilon }}_{VV}-{\mathrm{\Upsilon }}_{VW}{\mathrm{\Upsilon }}_{WW}^{-1}{\mathrm{\Upsilon }}_{WV}\end{array}\right),$$

we have, by Lemma 2 again, ${({\mathrm{\Theta }}_{ZZ}^{-1})}_{UV}=-{({\mathrm{\Theta }}_{ZZ}^{-1})}_{UU}{\mathrm{\Theta }}_{UV}{\mathrm{\Theta }}_{VV}^{-1}$. This can be rewritten as

$${\mathrm{\Upsilon }}_{UV}={\mathrm{\Upsilon }}_{UW}{\mathrm{\Upsilon }}_{WW}^{-1}{\mathrm{\Upsilon }}_{WV}=-({\mathrm{\Upsilon }}_{UU}-{\mathrm{\Upsilon }}_{UW}{\mathrm{\Upsilon }}_{WW}^{-1}{\mathrm{\Upsilon }}_{UW}){\mathrm{\Theta }}_{UV}{\mathrm{\Theta }}_{VV}^{-1}.$$ (38)

Therefore, Θ_UV = 0 iff ${\mathrm{\Upsilon }}_{UV}-{\mathrm{\Upsilon }}_{UW}{\mathrm{\Upsilon }}_{WW}^{-1}{\mathrm{\Upsilon }}_{WV}=0$.

Now let f ∈ 𝓐_U, g ∈ 𝓐_V. Then, by the definition of a projection,

$$\langle f-P_{{\mathcal{A}}_W}f,g-P_{{\mathcal{A}}_W}g\rangle =\langle f,g\rangle -\langle f,P_{{\mathcal{A}}_W}g\rangle .$$

The first term on the right is 〈f, ϒ_UVg〉_𝓐⊕X. Because, for any h ∈ 𝓐_W,

$$\langle h,{\mathrm{\Upsilon }}_{WW}^{-1}{\mathrm{\Upsilon }}_{WV}g\rangle ={\langle h,{\mathrm{\Upsilon }}_{WV}g\rangle }_{A_{\oplus X}}=\langle h,g\rangle ,$$

we have ${\mathrm{\Upsilon }}_{WW}^{-1}{\mathrm{\Upsilon }}_{WV}=P_{{\mathcal{A}}_W}\mid {\mathcal{A}}_V$. Hence

$$\langle f,P_{{\mathcal{A}}_W}g\rangle =\langle f,{\mathrm{\Upsilon }}_{WW}^{-1}{\mathrm{\Upsilon }}_{WV}g\rangle ={\langle f,{\mathrm{\Upsilon }}_{UW}{\mathrm{\Upsilon }}_{WW}^{-1}{\mathrm{\Upsilon }}_{WV}g\rangle }_{A_{\oplus X}}.$$

That is, we have shown that

$$\langle f-P_{{\mathcal{A}}_W}f,g-P_{{\mathcal{A}}_W}g\rangle ={\langle f,({\mathrm{\Upsilon }}_{UV}-{\mathrm{\Upsilon }}_{UW}{\mathrm{\Upsilon }}_{WW}^{-1}{\mathrm{\Upsilon }}_{WV})g\rangle }_{{\mathcal{A}}_{\oplus X}},$$

which means ${\mathrm{\Upsilon }}_{UV}-{\mathrm{\Upsilon }}_{UW}{\mathrm{\Upsilon }}_{WW}^{-1}{\mathrm{\Upsilon }}_{WV}=0$ iff U ⫫_A V|W.

Proof of Lemma 3

The proof of the three equalities are similar. So we only prove the last one. By (10), for any f ∈ 𝓐_X^−(i,j, g ∈ 𝓐_Xⁱ, we have

$${\langle f,{\mathrm{\sum }}_{X^{-(i,j)}{X}^i}g\rangle }_{{\mathcal{A}}_{X^{-(i,j)}}}=n^{-1}{[f]}^{\top }{K}_{-(i,j)}{QK}_{-(i,j)}^{\top }[{\mathrm{\sum }}_{X^{-(i,j)}{X}^i}][g].$$

Alternatively,

$${\langle f,{\mathrm{\sum }}_{X^{-(i,j)}{X}^i}g\rangle }_{A_{X^{-(i,j)}}}={cov}_n[f(X),g(X)]=n^{-1}{[f]}^{\top }{K}_{-(i,j)}{QK}_i[g].$$

Hence the right-hand sides of the above two equalities are the same for all [f] and [g], which implies the third relation in (11).

Proof of Proposition 4

1. Write (15) as V = LΨR^⊤ where L = (L₁, L₀), R = (R₁, R₀), and Ψ = diag(D, 0). Because L and R are orthogonal matrices,

   $${({VV}^{\top }+\epsilon I_s)}^{-1}={(L\mathrm{\Psi }{\mathrm{\Psi }}^{\top }{L}^{\top }+\epsilon I_s)}^{-1}=L{(\mathrm{\Psi }{\mathrm{\Psi }}^{\top }+\epsilon I_s)}^{-1}{L}^{\top }.$$ (39)
   By construction,

   $${(\mathrm{\Psi }{\mathrm{\Psi }}^{\top }+\epsilon I_s)}^{-1}={\{\text{diag}(D^2+\epsilon I_u,\epsilon I_{s-u})\}}^{-1}=\text{diag}\{{(D^2+\epsilon I_u)}^{-1},{\epsilon }^{-1}{I}_{s-u}\}.$$ (40)
   Hence the right-hand side of (39) is $L_1{(D^2+\epsilon I_u)}^{-1}{L}_1^{\top }+{\epsilon }^{-1}{L}_0{L}_0^{\top }$, which, because L is an orthogonal matrix, can be rewritten as

   $$L_1{(D^2+\epsilon I_u)}^{-1}{L}_1^{\top }+{\epsilon }^{-1}(I_s-L_1{L}_1^{\top })=L_1({(D^2+\epsilon I_u)}^{-1}-{\epsilon }^{-1}{I}_u)L_1^{\top }+{\epsilon }^{-1}{I}_s.$$

   By (15) we have $V=L_1{DR}_1^{\top }$, and hence L₁ = V R₁D⁻¹. Substitute this into the right-hand side to prove part 1.

2. Multiply both sides of (39) by V^⊤ from the left and by V from the right, to obtain

   $$V^{\top }{({VV}^{\top }+\epsilon I_s)}^{-1}V=R{\mathrm{\Psi }}^{\top }{(\top {\top }^{\top }+\epsilon I_s)}^{-1}\mathrm{\Psi }R.$$

   Substitute RΨ^⊤ = R₁D and (40) into the right-hand side above to prove part 2.

Proof of Theorem 7

Let f, g ∈ 𝓐_⊕X. Then f = f₁ + ··· + f_p and ϒg = h₁ + ··· + h_p where f_k, h_k ∈ 𝓐_X^k, k = 1, …, p. Hence

$${\langle f,\mathrm{\Upsilon }g\rangle }_{{\mathcal{A}}_{\oplus X}}=n^{-1}{\sum }_{i=1}^p{\langle f_i,h_i\rangle }_{{\mathcal{A}}_{X^i}}=n^{-1}{\sum }_{i=1}^p{[f_i]}^{\top }{K}_i{QK}_i[h_i]$$ (41)

Because

$${[f]}_{{\mathcal{A}}_{\oplus X}}={({[f_1]}_{{\mathcal{A}}_{X^1}}^{\top },\cdots ,{[f_p]}_{{\mathcal{A}}_{X^p}}^{\top })}^{\top },{[\mathrm{\Upsilon }g]}_{{\mathcal{A}}_{\oplus X}}={({[h_1]}_{{\mathcal{A}}_{X^1}}^{\top },\cdots ,{[h_p]}_{A_{X^p}}^{\top })}^{\top },$$

The right-hand side of (41) can be rewritten in matrix form as

$$n^{-1}{[f]}^{\top }\text{diag}(K_1{QK}_1,\cdots ,K_p{QK}_p)[\mathrm{\Upsilon }][g].$$ (42)

Alternatively, the inner product in (41) can be written as

$$\begin{array}{l}{\langle f,\mathrm{\Upsilon }g\rangle }_{{\mathcal{A}}_{\oplus X}}=n^{-1}{\sum }_{i=1}^p{\sum }_{j=1}^p{\langle f_i,{\mathrm{\sum }}_{X^i{X}^i}{g}_j\rangle }_{{\mathcal{A}}_{X^i}}\\ =n^{-1}{\sum }_{i=1}^p{\sum }_{j=1}^p{[f_i]}^{\top }{K}_i{QK}_j[g_j]=n^{-1}{[f]}^{\top }{K}_{1:p}{QK}_{1:p}^{\top }[g].\end{array}$$ (43)

Since (42) is equal to (43) for all [f] and [g], equality (16) holds.

Operator norm of CCO

By Fukumizu et al (2009),

$$[{\widehat{V}}_{X^i{X}^j\mid X^{-(i,j)}}]=n^{-1}{Q}_{X^{-(i,j)}}({\epsilon }_2)G_{X^i}.$$ (44)

The operator norm of V̂_{XⁱX^j|X^−(i,j)} is the maximum of 〈f_i, V̂_{XⁱX^j|X^−(i,j)} f_j〉² subject to 〈f_i, f_i〉 = 〈f_j, f_j〉 = 1. By linear algebra, this is equivalent to

$$\text{maximizing}\langle f_i,{\widehat{V}}_{X^i{X}^j\mid X^{-(i,j)}}{\widehat{V}}_{X^j{X}^i\mid X^{-(i,j)}}{f}_i\rangle \text{subject}\text{to}\langle f_i,f_i\rangle =1.$$ (45)

By (44), the first expression in (45) is n⁻²[f_i]^⊤G_Xⁱ Q_X^−(i,j) (ε₂)G_Xⁱ Q_X^−(i,j) (ε₂)G_Xⁱ [f_i]. The second expression in (45) is [f_i]^⊤ G_Xⁱ [f_i] = 1. This is a generalized eigenvalue problem. To convert this to an ordinary eigenvalue problem, let ${\varphi }_i=G_{X^i}^{1/2}[f_i]$. Solve this equation for [f_i] with Tychonoff regularization, to obtain [f_i] = (G_Xⁱ + ε₁I_n)^−1/2 ϕ_i. The optimization problem (45) reduces to maximizing (ignoring n⁻²):

$${\varphi }_i^{\top }{(G_{X^i}+{\epsilon }_1{I}_n)}^{-1/2}{G}_{X^i}{Q}_{X^{-(i,j)}}({\epsilon }_2)G_{X^j}{Q}_{X^{-(i,j)}}({\epsilon }_2)G_{X^i}{(G_{X^i}+{\epsilon }_1{I}_n)}^{-1/2}{\varphi }_i$$

subject to ${\varphi }_i^{\top }{\varphi }_i=1$. As we did with ACCO and APO, we use the symmetrized version

$${\varphi }_i^{\top }{(G_{X^i}+{\epsilon }_1{I}_n)}^{1/2}{Q}_{X^{-(i,j)}}({\epsilon }_2)(G_{X^j}+{\epsilon }_1{I}_n)Q_{X^{-(i,j)}}({\epsilon }_2){(G_{X^i}+{\epsilon }_1{I}_n)}^{1/2}{\varphi }_i,$$

which is of the form AA^⊤ with A being the matrix in (21).