Graphical Models via Univariate Exponential Family Distributions

Abstract

Undirected graphical models, or Markov networks, are a popular class of statistical models, used in a wide variety of applications. Popular instances of this class include Gaussian graphical models and Ising models. In many settings, however, it might not be clear which subclass of graphical models to use, particularly for non-Gaussian and non-categorical data. In this paper, we consider a general sub-class of graphical models where the node-wise conditional distributions arise from exponential families. This allows us to derive multivariate graphical model distributions from univariate exponential family distributions, such as the Poisson, negative binomial, and exponential distributions. Our key contributions include a class of M-estimators to fit these graphical model distributions; and rigorous statistical analysis showing that these M-estimators recover the true graphical model structure exactly, with high probability. We provide examples of genomic and proteomic networks learned via instances of our class of graphical models derived from Poisson and exponential distributions.

1. Introduction

Undirected graphical models, also known as Markov random fields, are an important class of statistical models that have been extensively used in a wide variety of domains, including statistical physics, natural language processing, image analysis, and medicine. The key idea in this class of models is to represent the joint distribution as a product of clique-wise compatibility functions. Given an underlying graph, each of these compatibility functions depends only on a subset of variables within any clique of the underlying graph. Popular instances of such graphical models include Ising and Potts models (see references in Wainwright and Jordan (2008) for a varied set of applications in computer vision, text analytics, and other areas with discrete variables), as well as Gaussian Markov Random Fields (GMRFs), which are popular in many scientific settings for modeling real-valued data. A key modeling question that arises, however, is: how do we pick the clique-wise compatibility functions, or alternatively, how do we pick the form or sub-class of the graphical model distribution (e.g. Ising or Gaussian MRF)? For the case of discrete random variables, Ising and Potts models are popular choices; but these are not best suited for count-valued variables, where the values taken by any variable could range over the entire set of positive integers. Similarly, in the case of continuous variables, Gaussian Markov Random Fields (GMRFs) are a popular choice; but the distributional assumptions imposed by GMRFs are quite stringent. The marginal distribution of any variable would have to be Gaussian for instance, which might not hold in instances when the random variables characterizing the data are skewed (Liu et al., 2009). More generally, Gaussian random variables have thin tails, which might not capture fat-tailed events and variables. For instance, in the finance domain, the lack of modeling of fat-tailed events and probabilities has been suggested as one of the causes of the 2008 financial crisis (Acemoglu, 2009).

To address this modeling question, some have recently proposed non-parametric extensions of graphical models. Some, such as the non-paranormal (Liu et al., 2009; Lafferty et al., 2012) and copula-based methods (Dobra and Lenkoski, 2011; Liu et al., 2012a), use or learn transforms that Gaussianize the data, and then fit Gaussian MRFs to estimate network structure. Others, use non-parametric approximations, such as rank-based estimators, to the correlation matrix, and then fit a Gaussian MRF (Xue and Zou, 2012; Liu et al., 2012b). More broadly, there could be non-parametric methods that either learn the sufficient statistics functions, or learn transformations of the variables, and then fit standard MRFs over the transformed variables. However, the sample complexity of such classes of non-parametric methods is typically inferior to those that learn parametric models. Alternatively, and specifically for the case of multivariate count data, Lauritzen (1996); Bishop et al. (2007) have suggested combinatorial approaches to fitting graphical models, mostly in the context of contingency tables. These approaches, however, are computationally intractable for even moderate numbers of variables.

Interestingly, for the case of univariate data, we have a good understanding of appropriate statistical models to use. In particular, a count-valued random variable can be modeled using a Poisson distribution; call-times, time spent on websites, diffusion processes, and life-cycles can be modeled with an exponential distribution; other skewed variables can be modeled with gamma or chi-squared distributions. Here, we ask if we can extend this modeling toolkit from univariate distributions to multivariate graphical model distributions? Interestingly, recent state of the art methods for learning Ising and Gaussian MRFs (Meinshausen and Bühlmann, 2006; Ravikumar et al., 2010; Jalali et al., 2011) suggest a natural procedure deriving such multivariate graphical models from univariate distributions. The key idea in these recent methods is to learn the MRF graph structure by estimating node-neighborhoods, or by fitting node-conditional distributions of each node conditioned on the rest of the nodes. Indeed, these node-wise fitting methods have been shown to have strong computational as well as statistical guarantees. Here, we consider the general class of models obtained by the following construction: suppose the node-conditional distributions of each node conditioned on the rest of the nodes follows a univariate exponential family. By the Hammersley-Clifford Theorem (Lauritzen, 1996), and some algebra as derived in Besag (1974), these node-conditional distributions entail a global multivariate distribution that (a) factors according to cliques defined by the graph obtained from the node-neighborhoods, and (b) has a particular set of compatibility functions specified by the univariate exponential family. The resulting class of MRFs, which we call exponential family MRFs, broadens the class of models available off the shelf, from the standard Ising, indicator-discrete, and Gaussian MRFs.

Thus the class of exponential family MRFs provides a principled approach to model multivariate distributions and network structures among a large number of variables; by providing a natural way to “extend” univariate exponential families of distributions to the multivariate case, in many cases where multivariate extensions did not exist in an analytical or computationally tractable form. Potential applications for these exponential family graphical models abound. Networks of call-times, time spent on websites, diffusion processes, and life-cycles can be modeled with exponential graphical models; other skewed multivariate data can be modeled with gamma or chi-squared graphical models; while multivariate count data such as from website visits, user-ratings, crime and disease incident reports, and bibliometrics could be modeled via Poisson graphical models. A key motivating application for our research is multivariate count data from next-generation genomic sequencing technologies (Mortazavi et al., 2008). This technology produces read counts of the number of short RNA fragments that have been mapped back to a particular gene; and measures gene expression with less technical variation than, and is thus rapidly replacing, microarrays (Marioni et al., 2008). Univariate count data is typically modeled using Poisson or negative binomial distributions (Li et al., 2011). As Gaussian graphical models have been traditionally used to understand genomic relationships and estimate regulatory pathways from microarray data, Poisson and negative-binomial graphical models could thus be used to analyze this next-generation sequencing data. Furthermore, there is a proliferation of new technologies to measure high-throughput genomic variation in which the data is not even approximately Gaussian (single nucleotide polymorphisms, copy number, methylation, and micro-RNA and gene expression via next-generation sequencing). For this data, a more general class of high-dimensional graphical models could thus lead to important breakthroughs in understanding genomic relationships and disease networks.

The construction of the class of exponential family graphical models also suggests a natural method for fitting such models: node-wise neighborhood estimation via sparsity constrained node-conditional likelihood maximization. A main contribution of this paper is to provide a sparsistency analysis (or analysis of variable selection consistency) for the recovery of the underlying graph structure of this broad class of MRFs. We note that the presence of non-linearities arising from the generalized linear models (GLM) posed subtle technical issues not present in the linear case (Meinshausen and Bühlmann, 2006). Indeed, for the specific cases of logistic, and multinomial respectively, Ravikumar et al. (2010); Jalali et al. (2011) derive such a sparsistency analysis via fairly extensive arguments, but which were tuned to the specific cases; for instance they used the fact that the variables were bounded, and the specific structure of the corresponding GLMs. Here we generalize their analysis to general GLMs, which required a subtler analysis as well as a slightly modified M-estimator. We note that this analysis might be of independent interest even outside the context of modeling and recovering graphical models. In recent years, there has been a trend towards unified statistical analyses that provide statistical guarantees for broad classes of models via general theorems (Negahban et al., 2012). Our result is in this vein and provides structure recovery for the class of sparsity constrained generalized linear models. We hope that the techniques we introduce might be of use to address the outstanding question of sparsity constrained M-estimation in its full generality.

There has been related work on the simple idea above of constructing joint distributions via specifying node-conditional distributions. Varin and Vidoni (2005); Varin et al. (2011) propose the class of composite likelihood models where the joint distribution is a function of the conditional distributions of subsets of nodes conditioned on other subsets. Besag (1974) discuss such joint distribution constructions in the context of node-conditional distributions belonging to exponential families, but for special cases of joint distributions such as pairwise models. In this paper, we consider the general case of higher-order graphical models for the joint distributions, and univariate exponential families for the node-conditional distributions. Moreover, a key contribution of the paper is that we provide tractable M-estimators with corresponding high-dimensional statistical guarantees and analysis for learning this class of graphical models even under high-dimensional statistical regimes.

Additionally, we note that a preliminary abridged version of this paper appeared at (Yang et al., 2012). In this manuscript, we provide a more in depth theoretical analysis along with several novel developments. Particularly, we provide a novel analytic framework on the sparsistency of our M-estimators that provide tighter finite-sample bounds, simpler proofs, and less restrictive assumptions than that of (Yang et al., 2012); these innovations are discussed further in Section 3.2. Further, we highlight and study several instances of our framework, relating our work to the existing literature on Gaussian MRFs and Ising models, as well as introducing two novel instances, the Poisson MRF and Exponential MRF. For each of these cases, we provide specific corollaries on conditions necessary for sparsistent recovery of the underlying graph structure. Finally, we also provide a greatly expanded experimental analysis of our class of MRFs and their M-estimators compared to that of (Yang et al., 2012). Focusing on two novel instances of our model, the Poisson and Exponential MRF, we study the theoretical rates, graph structural recovery, and robustness of our estimators through simulated examples. Further, we provide an additional case study on protein signaling networks using the Exponential MRF in Section 4.2.2.

2. Exponential Family Graphical Models

Suppose X = (X₁, …, X_p) is a random vector, with each variable X_i taking values in a set 𝒳. Let G = (V,E) be an undirected graph over the set of nodes V := {1, …, p} corresponding to the p variables ${\{X_r\}}_{r=1}^p$. The graphical model over X corresponding to G is a set of distributions that satisfy Markov independence assumptions with respect to the graph G (Lauritzen, 1996). By the Hammersley-Clifford theorem (Clifford, 1990), any such distribution that is strictly positive over its domain also factors according to the graph in the following way. Let 𝒞 be a set of cliques (fully-connected subgraphs) of the graph G, and let {ϕ_c(X_c)}_c∈𝒞 be a set of clique-wise sufficient statistics. With this notation, any strictly positive distribution of X within the graphical model family represented by the graph G takes the form:

$$P(X)\propto exp\{\sum _{c\in \mathcal{C}}{\theta }_c{\varphi }_c(X_c)\}$$ (1)

where {θ_c} are weights over the sufficient statistics. An important special case is a pairwise graphical model, where the set of cliques 𝒞 consists of the set of nodes V and the set of edges E, so that

$$P(X)\propto exp\{\sum _{r\in V}{\theta }_r{\varphi }_r(X_r)+\sum _{(r,t)\in E}{\theta }_{rt}{\varphi }_{rt}(X_r,X_t)\}.$$ (2)

As previously discussed, an important question is how to select the form of the graphical model distribution, which under the above parametrization in (1), translates to the question of selecting the class of sufficient statistics, ϕ. As discussed in the introduction, it is of particular interest to derive such a graphical model distribution as a multivariate extension of specified univariate parametric distributions such as negative binomial, Poisson, and others. We next outline a subclass of graphical models that answer these questions via the simple construction: set the node-conditional distributions of each node conditioned on the rest of the nodes as following a univariate exponential family, and then derive the joint distribution that is consistent with these node-conditional distributions. Then, in Section 3, we will study how to learn the underlying graph structure, or the edge set E, for this general class of “exponential family” graphical models. We provide a natural sparsity-encouraging M-estimator, and sufficient conditions under which the M-estimator recovers the graph structure with high probability.

2.1 The Form of Exponential Family Graphical Models

A popular class of univariate distributions is the exponential family, whose distribution for a random variable Z is given by

$$P(Z)=exp\{\theta B(Z)+C(Z)-D(\theta )\},$$ (3)

with sufficient statistics B(Z), base measure C(Z), and log-normalization constant D(θ). Such exponential family distributions include a wide variety of commonly used distributions such as Gaussian, Bernoulli, multinomial, Poisson, exponential, gamma, chi-squared, beta, and many others; any of which can be instantiated with particular choices of the functions B(·), and C(·). Such exponential family distributions are thus used to model a wide variety of data types including skewed continuous data and count data. Here, we ask if we can leverage this ability to model univariate data to also model the multivariate case. Let X = (X₁, X₂, …, X_p) be a p-dimensional random vector; and let G = (V,E) be an undirected graph over p nodes corresponding to the p variables. Could we then derive a graphical model distribution over X with underlying graph G, from a particular choice of univariate exponential family distribution (3) above?

Consider the following construction. Set the distribution of X_r given the rest of nodes X_V\r to be given by the above univariate exponential family distribution (3), and where the canonical exponential family parameter θ is set to a linear combination of k-th order products of univariate functions {B(X_t)}_t∈N(r), where N(r) is the set of neighbors of node r according to graph G. This gives the following conditional distribution:

$$P(X_r|X_{V\backslash r})=exp\{B(X_r)({\theta }_r+\sum _{t\in N(r)}{\theta }_{rt}B(X_t)+\sum _{t_2,t_3\in N(r)}{\theta }_{r{t}_2{t}_3}B(X_{t_2})B(X_{t_3})+\dots +\sum _{t_2,\dots ,t_k\in N(r)}{\theta }_{r{t}_2\dots t_k}\prod _{j=2}^{k}B(X_{t_j}))+C(X_r)-\overline{D}(X_{V\backslash r})\},$$ (4)

where C(X_r) is specified by the exponential family, and D̄(X_V\r) is the log-normalization constant. Notice that we use the notation D̄(·) in case when we express the log-partition function in terms of random variables. That is, D̄(X_V\r) := D(θ(X_V\r)) where θ(X_V\r) is the canonical parameter θ derived from X_V\r.

By the Hammersley-Clifford theorem, and some elementary calculation, this conditional distribution can be shown to specify the following unique joint distribution P(X₁, …, X_p):

Proposition 1 Suppose X = (X₁, X₂, …, X_p) is a p-dimensional random vector, and its node-conditional distributions are specified by (4) given an undirected graph G. Then its joint distribution P(X₁, …, X_p) belongs to the graphical model represented by G, and is given by

$$P(X)=exp\{\sum _{r\in V}{\theta }_{r}B(X_r)+\sum _{r\in V}\sum _{t\in N(r)}{\theta }_{rt}B(X_r)B(X_t)+\dots +\sum _{r\in V}\sum _{t_2,\dots ,t_k\in N(r)}{\theta }_{r\dots t_k}B(X_r)\prod _{j=2}^{k}B(X_{t_j})+\sum _{r\in V}C(X_r)-A(\theta )\}$$ (5)

where A(θ) is the log-normalization constant.

Note that the function D(·) (and hence D̄(·)) in (4) is the log-partition function of the node-conditional distribution, while the function A(·) in (5) in turn is the log-partition function of the joint distribution. Proposition 1, thus, provides an answer to our earlier question on selecting the form of a graphical model distribution given a univariate exponential family distribution. When the node-conditional distributions follow a univariate exponential family as in (4), there exists a unique graphical model distribution as specified by (5). One question that remains, however, is whether the above construction, beginning with (4), is the most general possible. In particular, note that the canonical parameter of the node-conditional distribution in (4) is a tensor factorization of the univariate sufficient statistic, which seems a bit stringent. Interestingly, by extending the argument from (Besag, 1974), which considers the special pairwise case, and the Hammersley-Clifford Theorem, we can show that indeed (4) and (5) have the most general form.

Theorem 2 Suppose X = (X₁, X₂, …, X_p) is a p-dimensional random vector, and its node-conditional distributions are specified by an exponential family,

$$P(X_r|X_{V\backslash r})=exp\{E(X_{V\backslash r})B(X_r)+C(X_r)-\overline{D}(X_{V\backslash r})\},$$ (6)

where the function E(X_V\r), the canonical parameter of exponential family, depends on the rest of all random variables except X_r (and hence the log-normalization constant D̄(X_V\r)). Further, suppose the corresponding joint distribution factors according to the graph G, with the factors over cliques of size at most k. Then, the conditional distribution in (6) necessarily has the tensor-factorized form in (4), and the corresponding joint distribution has the form in (5).

Theorem 2 thus tells us that under the general assumptions that:

1. the joint distribution is a graphical model that factors according to a graph G, and has clique-factors of size at most k, and

2. its node-conditional distribution follows an exponential family,

it necessarily follows that the conditional and joint distributions are given by (4) and (5) respectively.

An important special case is when the joint graphical model distribution has clique factors of size at most two. From Theorem 2, the conditional distribution is given by

$$P(X_r|X_{V\backslash r})=exp\{{\theta }_{r}B(X_r)+\sum _{t\in N(r)}{\theta }_{rt}B(X_r)B(X_t)+C(X_r)-\overline{D}(X_{V\backslash r})\},$$ (7)

while the joint distribution is given as

$$P(X)=exp\{\sum _{r\in V}{\theta }_{r}B(X_r)+\sum _{(r,t)\in E}{\theta }_{rt}B(X_r)B(X_t)+\sum _{r\in V}C(X_r)-A(\theta )\}.$$ (8)

For many classes of models (e.g. general Ising, discrete CRFs), the log-partition function of the joint distribution, A(·), has no analytical form, and might even be intractable to compute, while the function D(·) typically is more amenable, and available in analytical form, since it is the log-partition function of a univariate exponential family distribution.

When the univariate sufficient statistic function B(·) is a linear function B(X_r) = X_r, then the conditional distribution in (7) is precisely a generalized linear model (McCullagh and Nelder, 1989) in canonical form,

$$P(X_r|X_{V\backslash r})=exp\{{\theta }_r{X}_r+\sum _{t\in N(r)}{\theta }_{rt}{X}_r{X}_t+C(X_r)-\overline{D}(X_{V\backslash r};\theta )\},$$ (9)

where the canonical parameter of GLMs becomes θ_r + Σ_t∈N(r) θ_rt X_t. At the same time, the joint distribution has the form:

$$P(X)=exp\{\sum _{r\in V}{\theta }_r{X}_r+\sum _{(r,t)\in E}{\theta }_{rt}{X}_r{X}_t+\sum _{r\in V}C(X_r)-A(\theta )\}$$ (10)

where the log-partition function A(·) in this case is defined as

$$A(\theta ):=log{\int }_{X}exp\{\sum _{r\in V}{\theta }_r{X}_r+\sum _{(r,t)\in E}{\theta }_{rt}{X}_r{X}_t+\sum _{r\in V}C(X_r)\}dX.$$ (11)

We will now provide some examples of our general class of “exponential family” graphical model distributions, focusing on the case in (10) with linear functions B(X_r) = X_r. For each of these examples, we will also detail the domain, Θ := {θ : A(θ) < +∞}, of valid parameters that ensure that the density is normalizable. Indeed, such constraints on valid parameters are typically necessary for the distributions over countable discrete or continuous valued variables.

2.2 Gaussian Graphical Models

The popular Gaussian graphical model (Speed and Kiiveri, 1986) can be derived as an instance of the construction in Theorem 2, with the univariate Gaussian distribution as the exponential family distribution. The univariate Gaussian distribution with known variance σ² is given by

$$P(Z)\propto exp\{\frac{\mu }{\sigma }\frac{Z}{\sigma }-\frac{Z^2}{2{\sigma }^2}\}$$

where Z ∈ ℝ, so that it can be seen to be an exponential family distribution of the form (3), with sufficient statistic $B(Z)=\frac{Z}{\sigma }$, and base measure $C(Z)=-\frac{Z^2}{2{\sigma }^2}$. Substituting these in (10), we get the distribution:

$$P(X;\theta )\propto exp\{\sum _{r\in V}\frac{1}{{\sigma }_r}{\theta }_r{X}_r+\sum _{(r,t)\in E}\frac{1}{{\sigma }_r{\sigma }_t}{\theta }_{rt}{X}_r{X}_t-\sum _{r\in V}\frac{X_r^2}{2{\sigma }_r^2}\},$$ (12)

which can be seen to be the multivariate Gaussian distribution. Note that the set of parameters {θ_rt}_(r,t)∈E entails a precision matrix that needs to be positive definite for a valid probability distribution.

2.3 Ising Models

The Ising model (Wainwright and Jordan, 2008) in turn can be derived from the construction in Theorem 2 with the Bernoulli distribution as the univariate exponential family distribution. The Bernoulli distribution is a member of the exponential family of the form (3), with sufficient statistic B(X) = X, and base measure C(X) = 0, and with variables taking values in the set 𝒳 = {0, 1}. Substituting these in (10), we get the distribution:

$$P(X;\theta )=exp\{\sum _{(r,t)\in E}{\theta }_{rt}{X}_r{X}_t-A(\theta )\}$$ (13)

where we have ignored the singleton term, i.e. set θ_r = 0 for simplicity. The form of the multinomial graphical model, an extension of the Ising model, can also be represented by (10) and has been previously studied in Jalali et al. (2011) and others. The Ising model imposes no constraint on its parameters, {θ_rt}, for normalizability, since there are finitely many configurations of the binary random vector X.

2.4 Poisson Graphical Models

Poisson graphical models are an interesting instance with the Poisson distribution as the univariate exponential family distribution. The Poisson distribution is a member of the exponential family of the form (3), with sufficient statistic B(X) = X and C(X) = −log(X!), and with variables taking values in the set 𝒳 = {0, 1, 2, …}. Substituting these in (10), we get the following Poisson graphical model distribution:

$$P(X;\theta )=exp\{\sum _{r\in V}({\theta }_r{X}_r-log(X_r!))+\sum _{(r,t)\in E}{\theta }_{rt}{X}_r{X}_t-A(\theta )\}.$$ (14)

For this Poisson family, with some calculation, it can be seen that the normalizability condition, A(θ) < +∞, entails θ_rt ≤ 0 ∀ r, t. In other words, the Poisson graphical model can only capture negative conditional relationships between variables.

2.5 Exponential Graphical Models

Another interesting instance uses the exponential distribution as the univariate exponential family distribution, with sufficient statistic B(X) = −X and C(X) = 0, and with variables taking values in 𝒳 = {0} ⋃ ℝ⁺. Such exponential distributions are typically used for data describing inter-arrival times between events, among other applications. Substituting these in (10), we get the following exponential graphical model distribution:

$$P(X;\theta )=exp\{-\sum _{r\in V}{\theta }_r{X}_r-\sum _{(r,t)\in E}{\theta }_{rt}{X}_r{X}_t-A(\theta )\}.$$ (15)

To ensure that the distribution is valid and normalizable, so that A(θ) < +∞, we then require that θ_r > 0, θ_rt ≥ 0 ∀ r, t. Because of the negative sufficient statistic, this implies that the exponential graphical model can only capture negative conditional relationships between variables.

3. Statistical Guarantees on Learning Graphical Model Structures

In this section, we study the problem of learning the graph structure of an underlying exponential family MRF, given i.i.d. samples. Specifically, we assume that we are given n samples of random vector $X^{1:n}:={\{X^{(i)}\}}_{i=1}^n$, from a pairwise exponential family MRF,

$$P(X;{\theta }^{\ast })=exp\{\sum _{r\in V}{\theta }_r^{\ast }{X}_r+\sum _{(r,t)\in E^{\ast }}{\theta }_{rt}^{\ast }{X}_r{X}_t+\sum _{r}C(X_r)-A({\theta }^{\ast })\}.$$ (16)

The goal in graphical model structure recovery is to recover the edges E* of the underlying graph G = (V, E*). Following Meinshausen and Bühlmann (2006); Ravikumar et al. (2010); Jalali et al. (2011), we will approach this problem via neighborhood estimation: where we estimate the neighborhood of each node individually, and then stitch these together to form the global graph estimate. Specifically, if we have an estimate N̂(r) for the true neighborhood N*(r), then we can estimate the overall graph structure as

$$\widehat{E}={\cup }_{r\in V}{\cup }_{t\in \widehat{N}(r)}\{(r,t)\}.$$ (17)

Remark. Note that the node-neighborhood estimates N̂(r) might not be symmetric (i.e. there may be a pair (r, s) ∈ V × V, with r ∈ N̂(s), but s ∉ N̂(r)). The graph-structure estimate in (17) provides one way to reconcile these neighborhood estimates; see Meinshausen and Bühlmann (2006) for some other ways to do so (though as they note, these different estimates have asymptotically identical sparsistency guarantees: given exponential convergence in the probability of node-neighborhood recovery to one, the probability that the node-neighborhood estimates are symmetric, and hence that the different “reconciling” graph estimates would become identical, also converges to one.)

The problem of graph structure recovery can thus be reduced to the problem of recovering the neighborhoods of all the nodes in the graph. In order to estimate the neighborhood of any node in turn, we consider the sparsity constrained conditional MLE. Note that given the joint distribution in (16), the conditional distribution of X_r given the rest of the nodes is reduced to a GLM and given by

$$P(X_r|X_{V\backslash r})=exp\{X_r({\theta }_r^{\ast }+\sum _{t\in N^{\ast }(r)}{\theta }_{rt}^{\ast }{X}_t)+C(X_r)-D({\theta }_r^{\ast }+\sum _{t\in N^{\ast }(r)}{\theta }_{rt}^{\ast }{X}_t)\}.$$ (18)

Let θ*(r) be a set of parameters related to the node-conditional distribution of node X_r, i.e. ${\theta }^{\ast }(r)=({\theta }_r^{\ast },{\theta }_{\backslash r}^{\ast })\in ℝ\times {ℝ}^{p-1}$ where ${\theta }_{\backslash r}^{\ast }={\{{\theta }_{rt}^{\ast }\}}_{t\in V\backslash r}$ be a zero-padded vector, with entries ${\theta }_{rt}^{\ast }$ for t ∈ N*(r) and ${\theta }_{rt}^{\ast }=0$, for t ∉ N*(r). In order to infer the neighborhood structure for each node X_r, we solve the ℓ₁ regularized conditional log-likelihood loss:

$$\underset{\theta (r)\in \mathrm{\Omega }}{\text{minimize}}\{\mathcal{l}(\theta (r);X^{1:n})+{\lambda }_n{\Vert {\theta }_{\backslash r}\Vert }_1\}$$ (19)

where Ω is the parameter space in ℝ × ℝ^p−1, and ℓ(θ(r); X^1:n) is the conditional log-likelihood of the distribution (18):

$$\mathcal{l}(\theta (r);X^{1:n}):=-\frac{1}{n}log\prod _{i=1}^{n}P(X_r^{(i)}|X_{V\backslash r}^{(i)},\theta (r))=\frac{1}{n}\sum _{i=1}^n\{-X_r^{(i)}({\theta }_r+\langle {\theta }_{\backslash r},X_{V\backslash r}^{(i)}\rangle )+D({\theta }_r+\langle {\theta }_{\backslash r},X_{V\backslash r}^{(i)}\rangle )\}.$$

Note that the parameter space Ω might be restricted, and strictly smaller than ℝ × ℝ^p−1; for Poisson graphical models, θ_rt ≤ 0 for all r, t ∈ V for instance.

Given the solution θ̂(r) of the M-estimation problem above, we then estimate the node-neighborhood of r as N̂(r) = {t ∈ V\r : θ̂_rt ≠ 0}. In what follows, when we focus on a fixed node r ∈ V, we will overload notation, and use θ ∈ ℝ × ℝ^p−1 as the parameters of the conditional distribution, suppressing dependence on the node r.

3.1 Conditions

A key quantity in the analysis is the Fisher information matrix, $Q_r^{\ast }={\nabla }^2\mathcal{l}({\theta }^{\ast };X^{1:n})$, which is the Hessian of the node-conditional log-likelihood. In the following, we again will simply use Q* instead of $Q_r^{\ast }$ where the reference node r should be understood implicitly. We also use S = {(r, t) : t ∈ N*(r)} to denote the true neighborhood of node r, and S^c to denote its complement. We use $Q_{SS}^{\ast }$ to denote the d × d sub-matrix of Q* indexed by S where d is the number of neighborhoods of node r again suppressing dependence on r. Our first two conditions, mirroring those in Ravikumar et al. (2010), are as follows.

(C1) (Dependency condition) There exists a constant ρ_min > 0 such that ${\lambda }_{min}(Q_{SS}^{\ast })\ge {\rho }_{min}$ so that the sub-matrix of Fisher information matrix corresponding to true neighborhood has bounded eigenvalues. Moreover, there exists a constant ρ_max < ∞ such that ${\lambda }_{max}(\frac{1}{n}{\sum }_{i=1}^n[X_{V\backslash r}^{(i)}{(X_{V\backslash r}^{(i)})}^T])\le {\rho }_{max}$.

These condition can be understood as ensuring that variables do not become overly dependent. We will also need an incoherence or irrepresentable condition on the Fisher information matrix as in Ravikumar et al. (2010).

(C2) (Incoherence condition) There exists a constant α > 0, such that ${max}_{t\in S^c}{\Vert Q_{tS}^{\ast }{(Q_{SS}^{\ast })}^{-1}\Vert }_1\le 1-\alpha$.

This condition, standard in high-dimensional analyses, can be understood as ensuring that irrelevant variables do not exert an overly strong effect on the true neighboring variables.

A key technical facet of the linear, logistic, and multinomial models in Meinshausen and Bühlmann (2006); Ravikumar et al. (2010); Jalali et al. (2011), used heavily in their proofs, was that the random variables {X_r} there were bounded with high probability. Unfortunately, in the general exponential family distribution in (18), we cannot assume this explicitly. Nonetheless, we show that we can analyze the corresponding regularized M-estimation problems under the following mild conditions on the log-partition functions of the joint and node-conditional distributions.

(C3) (Bounded Moments) For all r ∈ V, the first and second moments are bounded, so that

$$\mathbb{E}[X_r]\le {\kappa }_m\text{and}\mathbb{E}[X_r^2]\le {\kappa }_v,$$

for some constants κ_m, κ_v. Further, the log-partition function A(·) of the joint distribution (16) satisfies:

$$\underset{u:|u|\le 1}{max}\frac{{\partial }^2}{\partial {\theta }_r^2}A({\theta }^{\ast }+u{e}_r)\le {\kappa }_h,$$

for some constant κ_h, and where e_r ∈ ℝ^p2 is an indicator vector that is equal to one at the index corresponding to θ_r, and zero everywhere else. Further, it holds that

$$\underset{\eta :|\eta |\le 1}{max}\frac{{\partial }^2}{\partial {\eta }^2}{\overline{A}}_r(\eta ;{\theta }^{\ast })\le {\kappa }_h,$$

where Ā_r(η; θ*) is a slight variant of (11):

$${\overline{A}}_r(\eta ;\theta ):=log{\int }_{X}exp\{\eta X_r^2+\sum _{u\in V}{\theta }_u{X}_u+\sum _{(u,t)\in V^2}{\theta }_{ut}{X}_u{X}_t+\sum _{u\in V}C(X_u)\}dX$$ (20)

for some scalar variable η.

(C4) For all r ∈ V, the log-partition function D(·) of the node-wise conditional distribution (18) satisfies: there exist functions κ₁(n, p) and κ₂(n, p) (that depend on the exponential family) such that, for all θ ∈ Θ and X ∈ 𝒳, |D″(a)| ≤ κ₁(n, p) where a ∈ [b, b+4κ₂(n, p) max{log n, log p}] for b := θ_r + 〈θ_\r, X_V\r〉. Additionally, |D‴(b)| ≤ κ₃(n, p) for all θ ∈ Θ and X ∈ 𝒳. Note that κ₁(n, p),κ₂(n, p) and κ₃(n, p) are functions that might be dependent on n and p, which affect our main theorem below.

Conditions (C3) and (C4) are the key technical components enabling us to generalize the analyses in Meinshausen and Bühlmann (2006); Ravikumar et al. (2010); Jalali et al. (2011) to the general exponential family case. It is also important to note that almost all exponential family distributions including all our previous examples can satisfy (C4) with mild functions κ₁(n, p), κ₂(n, p) and κ₃(n, p), as we will explicitly show later in this section. Comparing to the assumption in Yang et al. (2012) that requires ‖θ*‖₂ ≤ 1 for some exponential families, this will be much less restrictive condition on the minimum values of θ* permitted to achieve variable selection consistency.

3.2 Statement of the Sparsistency Result

Armed with the conditions above, we can show that the random vector X following a exponential family MRF distribution in (16) is suitably well-behaved:

Proposition 3 Suppose X is a random vector with the distribution specified in (16). Then, for ∀r ∈ V,

$$P(\frac{1}{n}\sum _{i=1}^n{(X_r^{(i)})}^2\ge \delta )\le exp(-cn{\delta }^2)$$

where δ ≤ min{2κ_v/3, κ_h + κ_v}, and c is a positive constant.

We recall the notation that the superscript indicates the sample and the subscript indicates the node; so that X⁽ⁱ⁾ is the i-th sample, while $X_s^{(i)}$ is the s-th variable/node of this random vector.

Proposition 4 Suppose X is a random vector with the distribution specified in (16). Then, for ∀r ∈ V,

$$P(|X_r|\ge \delta log\eta )\le c{\eta }^{-\delta }$$

where δ is any positive real value, and c is a positive constant.

These propositions are key to the following sparsistency result for the general family of pairwise exponential family MRFs (16).

Theorem 5 Consider a pairwise exponential family MRF distribution as specified in (16), with true parameter θ* and associated edge set E* that satisfies Conditions (C1)-(C4). Suppose that ${min}_{(s,t)\in E^{\ast }}|{\theta }_{rt}^{\ast }|\ge \frac{10}{{\rho }_{min}}\sqrt{d}{\lambda }_n$, where d is the maximum neighborhood size. Suppose also that the regularization parameter is chosen such that $M_1\frac{(2-\alpha )}{\alpha }\sqrt{{\kappa }_1(n,p)}\sqrt{\frac{logp}{n}}\le {\lambda }_n\le M_2\frac{(2-\alpha )}{\alpha }{\kappa }_1(n,p){\kappa }_2(n,p)$ for some constants M₁, M₂ > 0. Then, there exist positive constants L, c₁, c₂ and c₃ such that if n ≥ Ld²κ₁(n, p)(κ₃(n, p))² log p(max{log n, log p})², then with probability at least 1 – c₁(max{n, p})⁻² – exp(−c₂n) – exp(−c₃n), the following statements hold.

1. (Unique Solution) For each node r ∈ V, the solution of the M-estimation problem in (19) is unique, and

2. (Correct Neighborhood Recovery) The M-estimate also recovers the true neighborhood exactly, so that N̂(r) = N*(r).

Note that if the neighborhood of each node is recovered with high probability, then by a simple union bound, the estimate in (17), Ê = ⋃_r∈V ⋃_t∈N̂(r) {(r, t)} is equal to the true edge set E* with high-probability.

In the following subsections, we investigate the consequences of Theorem 5 for the sparsistency of specific instances of our general exponential family MRFs.

3.3 Statistical Guarantees for Gaussian MRFs, Ising Models, Exponential Graphical Models

In order to apply Theorem 5 to a specific instance of our general exponential family MRFs, we need to specify the terms κ₁(n, p), κ₂(n, p) and κ₃(n, p) defined in Condition (C4). It turns out that we can specify these terms for the Gaussian graphical models, Ising models and Exponential graphical model distributions, discussed in Section 2, in a similar manner, since the node-conditional log-partition function D(·) for all these distributions can be upper bounded by some constant independent of n and p. In particular, we can set κ₁(n, p) := κ₁, κ₂(n, p) := ∞ and κ₁(n, p) := κ₃ where κ₁ and κ₃ now become some constants depending on the distributions.

(Gaussian MRFs) Recall that the node-conditional distribution for Gaussian MRFs follow a univariate Gaussian distribution:

$$P(X_r|X_{V\backslash r})\propto exp\{X_r({\theta }_r+\sum _{t\in N(r)}{\theta }_{rt}{X}_t)-\frac{1}{2}{X}_r^2-\frac{1}{2}{({\theta }_r+\sum _{t\in N(r)}{\theta }_{rt}{X}_t)}^2\}.$$

Note that following (Meinshausen and Bühlmann, 2006), we assume that ${\sigma }_r^2=1$ for all r ∈ V. The node-conditional log-partition function D(·) can thus be written as $D(\eta ):=-\frac{1}{2}{\eta }^2$, so that |D″(η)| = 1 and D‴(η) = 0. We can thus set κ₁ = 1 and κ₃ = 0.

(Ising Models) For Ising models, node-conditional distribution follows a Bernoulli distribution:

$$P(X_r|X_{V\backslash r})=exp\{X_r(\sum _{t\in N(r)}{\theta }_{rt}{X}_t)-log(1+exp(\sum _{t\in N(r)}{\theta }_{rt}{X}_t))\}.$$

The node-conditional log-partition function D(·) can thus be written as D(η) := log (1 + exp(η)), so that for any η, $|D″(\eta )|=\frac{exp(\eta )}{{(1+exp(\eta ))}^2}\le \frac{1}{4}$ and $|D‴(\eta )|=\left|\frac{exp(\eta )(1-exp(\eta ))}{{(1+exp(\eta ))}^3}\right|<\frac{1}{4}$. Hence, we can set κ₁ = κ₃ = 1/4.

(Exponential Graphical Models) Lastly, for exponential graphical models, we have

$$P(X_r|X_{V\backslash r})=exp\{-X_r({\theta }_r+\sum _{t\in N(r)}{\theta }_{rt}{X}_t)+log({\theta }_r+\sum _{t\in N(r)}{\theta }_{rt}{X}_t)\}.$$

The node-conditional log-partition function D(·) can thus be written as D(η) := −log η, with η = θ_r + Σ_t∈N(r) θ_rtX_t. Recall from Section 2.5 that the node parameters are strictly positive θ_r > 0, and the edge-parameters are positive as well, θ_rt ≥ 0, as are the variables themselves X_t ≥ 0. Thus, under the additional constraint that θ_r > a₀ where a₀ is a constant smaller than ${\theta }_r^{\ast }$, we have that η := θ_r + Σ_t∈N(r) θ_rtX_t ≥ a₀. Consequently, $|D″(\eta )|=\frac{1}{{\eta }^2}\le \frac{1}{a_0^2}$ and $|D‴(\eta )|=\left|\frac{2}{{\eta }^3}\right|\le \frac{2}{a_0^3}$. We can thus set ${\kappa }_1=\frac{1}{a_0^2}$ and ${\kappa }_3=\frac{2}{a_0^3}$.

Armed with these derivations, we recover the following result on the sparsistency of Gaussian, Ising and Exponential graphical models, as a corollary of Theorem 5:

Corollary 6 Consider a Gaussian MRF (12) or Ising model (13) or Exponential graphical model (15) distribution with true parameter θ*, and associated edge set E*, and which satisfies Conditions (C1)-(C3). Suppose that ${min}_{(s,t)\in E^{\ast }}|{\theta }_{rt}^{\ast }|\ge \frac{10}{{\rho }_{min}}\sqrt{d}{\lambda }_n$. Suppose also that the regularization parameter is set so that $M\frac{(2-\alpha )}{\alpha }\sqrt{{\kappa }_1}\sqrt{\frac{logp}{n}}\le {\lambda }_n$ for some constant M > 0. Then, there exist positive constants L, c₁, c₂ and c₃ such that if $n\ge L{\kappa }_1{\kappa }_3^2{d}^{2}logp{(max\{logn,logp\})}^2$, then with probability at least 1 –c₁ (max{n, p})⁻²–exp(−c₂n) – exp(−c₃n), the statements on the uniqueness of the solution and correct neighborhood recovery, in Theorem 5 hold.

Remarks. As noted, our models and theorems are quite general, extending well beyond the popular Ising and Gaussian graphical models. The graph structure recovery problem for Gaussian models was studied in Meinshausen and Bühlmann (2006) especially for the regime where the neighborhood sparsity index is sublinear, meaning that d/p → 0. Besides the sublinear scaling regime, Corollary 6 can be adapted to entirely different types of scaling, such as the linear regime where d/p → α for some α > 0 (see Wainwright (2009) for details on adaptations to sublinear scaling regimes). Moreover, with κ₁ and κ₃ as defined above, Corollary 6 exactly recovers the result in Ravikumar et al. (2010) for the Ising models as a special case.

Also note that Corollary 6 provides tighter finite-sample bounds than the results of Yang et al. (2012). In particular, a sample size complexity necessary on λ_n to achieve sparsistent recovery here is $O(\sqrt{\frac{logp}{n}})$, which is faster as compared to $O(\sqrt{\frac{logp}{n^{1-\kappa }}})$ in Yang et al. (2012).

3.4 Statistical Guarantees for Poisson Graphical Models

We now consider the Poisson graphical model. Again, to derive the corresponding corollary of Theorem 5, we need to specify the terms κ₁(n, p), κ₂(n, p) and κ₃(n, p) defined in Condition (C4). Recall that the node-conditional distribution of Poisson graphical models has the form:

$$P(X_r|X_{V\backslash r})=exp\{X_r({\theta }_r+\sum _{t\in N(r)}{\theta }_{rt}{X}_t)-log(X_r!)-exp({\theta }_r+\sum _{t\in N(r)}{\theta }_{rt}{X}_t)\}.$$

The node-conditional log-partition function D(·) can thus be written as D(η) := exp η, with η = θ_r + Σ_t∈N(r) θ_rtX_t. Noting that the variables {X_t} range over positive integers, and that feasible parameters θ_rt are negative, we obtain

$$D″(\eta )=D″({\theta }_r+\langle {\theta }_{\backslash r},X_{V\backslash r}\rangle +4{\kappa }_2(n,p)logp\prime )=exp({\theta }_r+\langle {\theta }_{\backslash r},X_{V\backslash r}\rangle +4{\kappa }_2(n,p)logp\prime )\le exp({\theta }_r+4{\kappa }_2(n,p)logp\prime )$$

where p′ = max{n, p}. Suppose that we restrict our attention on the subfamily where θ_r ≤ a₀ for some positive constant a₀. Then, if we choose κ₂(n, p) := 1/(4 log p′), we then obtain θ_r + 4κ₂(n, p) log p′ ≤ a₀ + 1, so that setting κ₁(n, p) := exp(a₀ + 1) would satisfy Condition (C4). Similarly, we obtain D‴(θ_r + 〈θ_\r, X_V\r〉) = exp (θ_r + 〈θ_\r, X_V\r〉) ≤ exp(a₀ + 1), so that we can set κ₃(n, p) to exp(a₀ + 1).

Armed with these settings, we recover the following corollary for Poisson graphical models:

Corollary 7 Consider a Poisson graphical model distribution as specified in (14), with true parameters θ*, and associated edge set E*, that satisfies Conditions (C1)-(C3). Suppose that ${min}_{(s,t)\in E^{\ast }}|{\theta }_{rt}^{\ast }|\ge \frac{10}{{\rho }_{min}}\sqrt{d}{\lambda }_n$. Suppose also that the regularization parameter is chosen such that $M_1\frac{(2-\alpha )}{\alpha }\sqrt{{\kappa }_1}\sqrt{\frac{logp}{n}}\le {\lambda }_n\le M_2{\kappa }_1\frac{(2-\alpha )}{\alpha }\frac{1}{max\{logn,logp\}}$ for some constants M₁, M₂ > 0. Then, there exist positive constants L, c₁, c₂ and c₃ such that if $n\ge L{d}^2{\kappa }_1{\kappa }_3^{2}logp{(max\{logn,logp\})}^2$, then with probability at least 1 – c₁(max{n, p})⁻² – exp(−c₂n) – exp(−c₃n), the statements on the uniqueness of the solution and correct neighborhood recovery, in Theorem 5 hold.

4. Experiments

We evaluate our M-estimators for exponential family graphical models, specifically for the Poisson and exponential distributions, through simulations and real data examples. Neighborhood selection was performed for each M-estimator with an ℓ₁ penalty to induce sparsity and non-negativity or non-positivity constraints to enforce appropriate restrictions on the parameters. Optimization algorithms were implemented using projected gradient descent (Daubechies et al., 2008; Beck and Teboulle, 2010), which since the objectives are convex, is guaranteed to converge to the global optimum. Further details on the optimization problems used for our M-estimators are given in the Appendix E.

4.1 Simulation Studies

We provide a small simulation study that corroborates our sparsistency results; specifically Corollary 6 for the exponential graphical model, where node-conditional distributions follow an exponential distribution, and Corollary 7 for the Poisson graphical model, where node-conditional distributions follow a Poisson distribution. We instantiated the corresponding exponential and Poisson graphical model distributions in (15) and (14) for 4 nearest neighbor lattice graphs (d = 4), with varying number of nodes, p ∈ {64, 100, 169, 225}, and with identical edge weights for all edges: for exponential MRF, ${\theta }_r^{\ast }=0.1$ and ${\theta }_{rt}^{\ast }=1$, and, for Poisson MRF, ${\theta }_r^{\ast }=2$ and ${\theta }_{rt}^{\ast }=-0.1$. We generated i.i.d. samples from these distributions using Gibbs sampling, and solved our sparsity-constrained M-estimation problem by setting ${\lambda }_n=c\sqrt{\frac{logp}{n}}$, following our corollaries; c = 3 for exponential MRF, and 15 for Poisson MRF. We repeated each simulation 50 times and measured the empirical probability over the 50 trials that our penalized graph estimate in (17) successfully recovered all edges, that is, P(Ê = E*). The left panels of Figure 1(a) and Figure 1(b) show the empirical probability of successful edge recovery. In the right panel, we plot the empirical probability against a re-scaled sample size β = n/(log p). According to our corollaries, the sample size n required for successful graph structure recovery scales logarithmically with the number of nodes p. Thus, we would expect the empirical curves for different problem sizes to more closely align with this re-scaled sample size on the horizontal axis, a result clearly seen in the right panels of Figure 1. This small numerical study thus corroborates our theoretical sparsistency results.

Figure 1.

Probabilities of successful support recovery for the (a) exponential MRF, grid structure with parameters ${\theta }_r^{\ast }=0.1$ and ${\theta }_{rt}^{\ast }=1$, and the (b) Poisson MRF, grid structure with parameters ${\theta }_r^{\ast }=2$ and ${\theta }_{rt}^{\ast }=-0.1$. The empirical probability of successful edge recovery over 50 replicates is shown versus the sample size n (left), and verses the re-scaled sample size β = n/(log p) (right). The empirical curves align for the latter, thus verifying the logarithmic dependence of n on p as obtained in our sparsistency analysis.

We also evaluate the comparative performance of our M-estimators for recovering the true edge structure from the different types of data. Specifically, we consider the three typical examples in our unified neighborhood selection approach: the Poisson M-estimator, the Exponential M-estimator, and the well-known Gaussian M-estimator by (Meinshausen and Bühlmann, 2006). In order to extensively compare their performances, we compute the receiver-operator-curves for the overall graph recovery by varying the regularization parameter, λ_n. In Figure 2, the same graph structures for the exponential MRF ( ${\theta }_r^{\ast }=0.1$ and ${\theta }_{rt}^{\ast }=1$) and the Poisson MRF ( ${\theta }_r^{\ast }=2$ and ${\theta }_{rt}^{\ast }=-0.1$) with 4 nearest neighbors, are used as in the previous simulation. Moreover, we focus on the high-dimensional regime where n < p. As shown in the figure, exponential and Poisson M-estimators outperform and have significant advantage over Gaussian neighborhood selection approach if the data is generated according to exponential or Poisson MRFs. One interesting phenomenon we observe is that exponential and Poisson M-estimators perform similarly regardless of the underlying graphical model distribution. This likely occurs as our estimator maximizes the conditional likelihoods by fitting penalized GLMs. Note that GLMs assume that the conditional mean of the regression model follows an exponential family distribution. As both the Poisson distribution and the exponential distribution have the same mean, the rate parameter, λ, we would expect GLM-based methods that fit conditional means to perform similarly.

Figure 2.

Receiver-operator curves (ROC) computed by varying the regularization parameter, λ_n. High-dimensional data is generated according to (a) the Exponential MRF with (n, p) = (150, 225) and to (b) the Poisson MRF with (n, p) = (100, 225). Results are compared for three M-estimators: that of the Poisson, exponential, and Gaussian distributions.

As discussed at end of Section 2, the exponential and Poisson graphical models are able to capture only negative conditional dependencies between random variables, and our corresponding M-estimators are computed under this constraint. In our last simulation, we evaluate the impact of this restriction when the true graph contains both positive and negative edge weights. As there does not exist a proper MRF related to the Poisson and exponential distributions with both positive and negative dependencies, we resort to generating data from via a copula transform. In particular, we first generate multivariate Gaussian samples from N(0, Σ) where Θ = Σ⁻¹ is the precision matrix corresponding to the 4 nearest neighbor grid structure previously considered. Specifically, Θ has all ones on the diagonal and ${\theta }_{rt}^{\ast }=\pm 0.2$ with equal probabilities. We then use a standard copula transform to make the marginals of the generated data approximately Poisson. Figure 3 again present receiver operator curves (ROC) for the three different classes of M-estimators on the copula transformed data, transformed to the Poisson distribution. In the left of Figure 3, we consider signed support recovery where we define the true positive rate as $\frac{\#\text{of edges s.t. sign}({\theta }_{rt}^{\ast })=\text{sign}({\widehat{\theta }}_{rt})}{\#\text{of edges}}$. In the right, on the other hand, we ignore the positive edges so that true positive rate is now $\frac{\#\text{of edges s.t. sign}({\theta }_{rt}^{\ast })=\text{sign}({\widehat{\theta }}_{rt})=-1}{\#\text{of negative edges}}$. Note that the false positive rate is also defined similarly. As expected, the results indicate that our Poisson and exponential M-estimators fail to recover the edges with positive conditional dependencies recovered by the Gaussian M-estimator. However, when attention is restricted to negative conditional dependencies, our method outperforms the Gaussian M-estimator. Notice also that for the exponential and Poisson M-estimators, the highest false positive rate achieved is around 0.15. This likely occurs due to the constraints enforced by our M-estimators that force the weights of potential positive conditional dependent edges to be zero. Thus, while the restrictions on the edge weights may be severe, for the purpose of estimating negative conditional dependencies with limited false positives, the Poisson and exponential M-estimators have an advantage.

Figure 3.

Receiver-operator curves (ROC) computed by varying the regularization parameter, λ_n, for data, (n, p) = (200, 225), generated via Poisson copula transform according to a network with both positive and negative conditional dependencies. Left plot denotes results on overall edge recovery, while right plot denotes recovery of the edges with negative weights corresponding to negative conditional dependencies.

4.2 Real Data Examples

To demonstrate the versatility of our family of graphical models, we also provide two real data examples: a meta-miRNA inhibitory network estimated by the Poisson graphical model, Figure 4, and a cell signaling network estimated by the exponential graphical model, Figure 5.

Figure 4.

Meta-miRNA inhibitory network for breast cancer estimated via Poisson graphical models from miRNA-sequencing data. Level III data from TCGA was processed into tightly correlated clusters, meta-miRNAs, with the driver miRNAs identified for each cluster taken as the set of nodes for our network. The Poisson network reveals major inhibitory relationships between three hub miRNAs, two of which have been previously identified as tumor suppressors in breast cancer.

Figure 5.

Cell signaling network estimated from flow cytometry data via exponential graphical models (left) and Gaussian graphical models (right). The exponential graphical model was fit to un-transformed flow cytometry data measuring 11 proteins, and the Gaussian graphical model to log-transformed data. Estimated negative conditional dependencies are given in red. Both networks identify PKA (protein kinase A) as a major inhibitor, consistent with previous results.

When applying our family of graphical models, there is always a question of whether our model is an appropriate fit for the observed data. Typically, one can assess model fit using goodness-of-fit tests. For the Gaussian graphical model, this reduces to testing whether the data follows a multivariate Gaussian distribution. For general exponential family graphical models, testing for goodness-of-fit is more challenging. Some have proposed likelihood ratio tests specifically for lattice systems with a fixed and known dependence structure (Besag, 1974). When the network structure is unknown, however, there are no such existing tests. While we leave the development of an exact test to future work, we provide a heuristic that can help us understand whether our model is appropriate for a given dataset.

Recall that our model assumes that conditional on its node-neighbors, each variable is distributed according to an exponential family. Thus, if the neighborhood is known, our conditional models are simply GLMs, for which the goodness-of-fit can be assessed compared to a null model by a likelihood ratio test (McCullagh and Nelder, 1989). When neighborhoods must be estimated, and specifically when estimated via an ℓ₁-norm penalty, the resulting ratio of likelihoods no longer follow a chi-squared distribution (Bühlmann, 2011). Recently, for the ℓ₁ linear regression case, Lockhart et al. (2014) have shown that the difference in the residual sums of squares follows an exponential distribution. Similar results have not yet been extended to the penalized GLM case. In the absence of such tests, we propose a simple heuristic: for each node, first estimate the node-neighborhood via our proposed M-estimator. Next, assuming the neighborhood is fixed, fit a GLM and compare the fit of this model to that of a null model (only an intercept term) via the likelihood ratio test. One can then heuristically assess the overall goodness-of-fit by examining the fit of a GLM to all the nodes. This procedure is clearly not an exact test, and following from Lockhart et al. (2014), it is likely conservative. In the absence of an exact test, which we leave for future work, this heuristic provides some assurances about the appropriateness of our model for real data.

4.2.1 Poisson Graphical Model: Meta-miRNA Inhibitory Network

Gaussian graphical models have often been used to study high-throughput genomic networks estimated from microarray data (Pe'er et al., 2001; Friedman, 2004; Wei and Li, 2007). Many high-throughput technologies, however, do not produce even approximately Gaussian data, so that our class of graphical models could be particularly important for estimating genomic networks from such data. We demonstrate the applicability of our class of models by estimating a meta-miRNA inhibitory network for breast cancer estimated by a Poisson graphical model. Level III breast cancer miRNA expression (Cancer Genome Atlas Research Network, 2012) as measured by next generation sequencing was downloaded from the TCGA portal (http://tcga-data.nci.nih.gov/tcga/). MicroRNAs (miRNA) are short RNA fragments that are thought to be post-transcriptional regulators, predominantly inhibiting translation. Measuring miRNA expression by high-throughput sequencing results in count data that is zero-inflated, highly skewed, and whose total count volume depends on experimental conditions (Li et al., 2011). Data was processed to be approximately Poisson by following the steps described in (Allen and Liu, 2013). In brief, the data was quantile corrected to adjust for sequencing depth (Bullard et al., 2010); the miRNAs with little variation across the samples, the bottom 50%, were filtered out; and the data was adjusted for possible over-dispersion using a power transform and a goodness of fit test (Li et al., 2011). We also tested for batch effects in the resulting data matrix consisting of 544 subjects and 262 miRNAs: we fit a Poisson ANOVA model (Leek et al., 2010), and only found 4% of miRNAs to be associated with batch labels; and thus no significant batch association was detected. As several miRNAs likely target the same gene or genes in the same pathway, we expect there to be strong positive dependencies among variables that cannot be captured directly by our Poisson graphical model which only permits negative conditional relationships. Thus, we will use our model to study inhibitory relationships between what we term meta-miRNAs, or groups of miRNAs that are tightly positively correlated. To accomplish this, we further processed our data to form clusters of positively correlated miRNAs using hierarchical clustering with average linkage and one minus the correlation as the distance metric. This resulted in 40 clusters of tightly positively correlated miRNAs. The nodes of our meta-miRNA network were then taken as a the medoid, or median centroid defined as the miRNA closest in Euclidean distance to the cluster centroid, in each group.

A Poisson graphical model was fit to the meta-miRNA data by performing neighborhood selection with the sparsity of the graph determined by stability selection (Liu et al., 2010). The heuristic previously discussed was used to assess goodness-of-fit for our model. Out of the 40 node-neighborhoods tested via a likelihood ratio test, 36 exhibited p-values less than 0.05, and 34 were less than 0.05/40, the Bonferroni-adjusted significance level. These results show that the Poisson GLM is a significantly better fit for the majority of node-neighborhoods than the null model, indicating that our Poisson graphical model is appropriate for this data. The results of our estimated Poisson graphical model, Figure 4 (left), are consistent with the cancer genomics literature. First, the meta-miRNA inhibitory network has three major hubs. Two of these, miR-519 and miR-520, are known to be breast cancer tumor suppressors, suppressing growth by reducing HuR levels (Abdelmohsen et al., 2010) and by targeting NF-KB and TGF-beta pathways (Keklikoglou et al., 2012) respectively. The third major hub, miR-3156, is a miRNA of unknown function; from its major role in our network, we hypothesize that miR-3156 is also associated with tumor suppression. Also interestingly, let-7, a well-known miRNA involved in tumor metastasis (Yu et al., 2007), plays a central role in our network, sharing edges with the five largest hubs. This suggests that our Poisson graphical model has recovered relevant negative relationships between miRNAs with the five major hubs acting as suppressors, and the central let-7 miRNA and those connected to each of the major hubs acting as enhancers of tumor progression in breast cancer.

4.2.2 Exponential Graphical Model: Inhibitory Cell-Signaling Network

We demonstrate our exponential graphical model, derived from the univariate exponential distribution, using a protein signaling example (Sachs et al., 2005). Multi-florescent flow cytometry was used to measure the presence of eleven proteins (p = 11) in n = 7462 cells. This data set was first analyzed using Bayesian Networks in Sachs et al. (2005) and then using the graphical lasso algorithm in Friedman et al. (2007). Measurements from flow-cytometry data typically follow a left skewed distribution. Thus to model such data, these measurements are typically normalized to be approximately Gaussian using a log transform after shifting the data to be non-negative (Herzenberg et al., 2006). Here, we demonstrate the applicability of our exponential graphical models to recover networks directly from continuous skewed data, so that we learn the network directly from the flow-cytometry data without any log or such transforms. Our pre-processing is limited to shifting the data for each protein so that it consists of non-negative values. For comparison purposes, we also fit a Gaussian graphical model to the log-transformed data.

We then learned an exponential and Gaussian graphical model from this flow cytometry data using stability selection (Liu et al., 2010) to select the sparsity of the graphs. The goodness-of-fit heuristic previously described was used to assess the appropriateness of our model. Out of the eight connected node-neighborhoods, the likelihood ratio test was statistically significant for seven neighborhoods, indicating that our exponential GLM is a better fit than the null model. The estimated protein-signaling network is shown on the right in Figure 5 with that of the Gaussian graphical model fit to the log-transformed data on the left. Estimated negative conditional dependencies are shown in red. Recall that the exponential graphical model restricts the edge weights to be non-negative; because of the negative inverse link, this implies that only negative conditional associations can be estimated. Notice that our exponential graphical model finds that PKA, protein kinase A, is a major protein inhibitor in cell signaling networks. This is consistent with the inhibitory relationship of PKA as estimated by the Gaussian graphical model, right Figure 5, as well as its hub status in the Bayesian network of (Sachs et al., 2005). Interestingly, our exponential graphical model also finds a clique between PIP2, Mek, and P38, which was not found by Gaussian graphical models.

5. Discussion

We study what we call the class of exponential family graphical models that arise when we assume that node-wise conditional distributions follow exponential family distributions. Our work broadens the class of off-the-shelf graphical models from classical instances such as Ising and Gaussian graphical models. In particular, our class of graphical models provide closed form multivariate densities as extensions of several univariate exponential family distributions (e.g. Poisson, exponential, negative binomial) where few currently exist; and thus may be of further interest to the statistical community. Further, we provide simple M-estimators for estimating any of these graphical models from data, by fitting node-wise penalized conditional exponential family distributions, and show that these estimators enjoy strong statistical guarantees. The statistical analyses of our M-estimators required subtle techniques that may be of general interest in the analysis of sparse M-estimation.

There are many avenues of future work related to our proposed models. We assume that all conditional distributions are members of an exponential family. To determine whether this assumption is appropriate in practice for real data, a goodness-of-fit procedure is needed. While we have proposed a heuristic to this effect, more work is needed to determine a rigorous likelihood ratio test for testing model fit. For several instances of our proposed class of models, specifically those with variables with infinite domains, severe restrictions on the parameter space are sometimes needed. For instance, the Poisson and exponential graphical models studied in Section 4, could only model negative conditional dependencies, which may not always be desirable in practice. A key question for future work is whether these restrictions can be relaxed for particular exponential family distributions. Finally, while we have focused on single parameter exponential families, it would be interesting to investigate the consequences of using multi-parameter exponential family distributions. Overall, our work has opened avenues for learning Markov networks from a broad class of univariate distributions, the properties and applications of which leave much room for future research.

Appendix A. Proof of Theorem 2

The proof follows the development in Besag (1974), where they consider the case with k = 2. We define Q(X) as Q(X) := log(P(X)/P(0)), for any X = (X₁, …, X_p) ∈ 𝒳^p where P(0)) denotes the probability that all random variables take 0. Given any X, also denote X̄_r:0 := (X₁, …, X_r−1, 0, X_r+1, …, X_p). Now, consider the following the most general form for Q(X):

$$Q(X)=\sum _{1\le r\le p}{X}_r{G}_r(X_r)+\dots +\sum _{1\le r_1<r_2<\dots <r_k\le p}{X}_{r_1}\dots X_{r_k}{G}_{r_1\dots r_k}(X_{r_1},\dots ,X_{r_k}),$$ (21)

since the joint distribution has factors of at most size k. By the definition of Q and some algebra (See Section 2 of Besag (1974) for details), it can then be seen that

$$exp(Q(X)-Q({\overline{X}}_{r:0}))=P(X_r|X_1,\dots ,X_{r-1},X_{r+1},\dots ,X_p)/P(0|X_1,\dots ,X_{r-1},X_{r+1},\dots ,X_p).$$ (22)

Now, consider the simplifications of both sides of (22). For notational simplicity, we fix r = 1 for a while. Given the form of Q(X) in (21), we have

$$Q(X)-Q({\overline{X}}_{1:0})=X_1(G_1(X_1)+\sum _{2\le t\le p}{X}_t{G}_{1t}(X_1,X_t)+\dots +\sum _{2\le t_2<t_3<\dots <t_k\le p}{X}_{t_2}\dots X_{t_k}{G}_{1t_2\dots t_k}(X_1,X_{t_2}\dots ,X_{t_k})).$$ (23)

By given the exponential family form of the node-conditional distribution specified in the statement, right-hand side of (22) become

$$log\frac{P(X_1|X_2,\dots ,X_p)}{P(0|X_2,\dots ,X_p)}=E(X_{V\backslash 1})(B(X_1)-B(0))+(C(X_1)-C(0)).$$ (24)

Setting X_t = 0 for all t ≠ 1 in (23) and (24), we obtain

$$X_1{G}_1(X_1)=E(\mathbf{0})(B(X_1)-B(0))+(C(X_1)-C(0)).$$

Setting X_t2 = 0 for all t₂ ∉ {1, t},

$$X_1{G}_1(X_1)+X_1{X}_t{G}_{1t}(X_1,X_t)=E(0,\dots ,X_t,\dots ,0)(B(X_1)-B(0))+(C(X_1)-C(0)).$$

Recovering the index 1 back to r yields

$$\begin{array}{l}{X}_r{G}_r(X_r)=E(\mathbf{0})(B(X_r)-B(0))+(C(X_r)-C(0)),\hfill \\ X_r{G}_r(X_r)+X_r{X}_t{G}_{rt}(X_r,X_t)=E(0,\dots ,X_t,\dots ,0)(B(X_r)-B(0))+(C(X_r)-C(0)).\hfill \end{array}$$

Similarly,

$$X_t{G}_t(X_t)+X_r{X}_t{G}_{rt}(X_r,X_t)=E(0,\dots ,X_r,\dots ,0)(B(X_t)-B(0))+(C(X_t)-C(0)).$$ (25)

From the above three equations, we obtain

$$X_r{X}_t{G}_{rt}(X_r,X_t)={\theta }_{rt}(B(X_r)-B(0))(B(X_t)-B(0)).$$

More generally, by considering non-zero triplets, and setting X_v = 0 for all v ∉ {r, t, u}, we obtain

$$X_r{G}_r(X_r)+X_r{X}_t{G}_{rt}(X_r,X_t)+X_r{X}_u{G}_{ru}(X_r,X_u)+X_r{X}_t{X}_u{G}_{rtu}(X_r,X_t,X_u)=E(0,\dots ,X_t,\dots ,X_u,\dots ,0)(B(X_r)-B(0))+(C(X_r)-C(0)),$$ (26)

so that by a similar reasoning we can obtain

$$X_r{X}_t{X}_u{G}_{rtu}(X_r,X_t,X_u)={\theta }_{rtu}(B(X_r)-B(0))(B(X_t)-B(0))(B(X_u)-B(0)).$$

More generally, we can show that

$$X_{t_1}\dots X_{t_k}{G}_{t_1,\dots ,t_k}(X_{t_1},\dots ,X_{t_k})={\theta }_{t_1,\dots ,t_k}(B(X_{t_1})-B(0))\dots (B(X_{t_k})-B(0)).$$

Thus, the k-th order factors in the joint distribution as specified in (21) are tensor products of (B(X_r) – B(0)), thus proving the statement of the theorem.

Appendix B. Proof of Proposition 3

By the definition of (20) with the following simple calculation, the moment generating function of $X_r^2$ becomes

$$log\mathbb{E}[exp(a{X}_r^2)]=log{\int }_{X}exp\{a{X}_r^2+\sum _{t\in V}{\theta }_t^{\ast }{X}_t+\sum _{(t,u)\in E}{\theta }_{tu}^{\ast }{X}_t{X}_u+\sum _{t\in V}C(X_t)-A({\theta }^{\ast })\}={\overline{A}}_r(a;{\theta }^{\ast })-{\overline{A}}_r(0;{\theta }^{\ast }).$$

Suppose that a ≤ 1. Then, by a Taylor Series expansion, we have for some ν ∈ [0, 1]

$${\overline{A}}_r(a;{\theta }^{\ast })-{\overline{A}}_r(0;{\theta }^{\ast })=a\frac{\partial }{\partial \eta }{\overline{A}}_r(0;{\theta }^{\ast })+\frac{1}{2}{a}^2\frac{{\partial }^2}{\partial {\eta }^2}{\overline{A}}_r(\nu a;{\theta }^{\ast })\le {\kappa }_{v}a+\frac{1}{2}{\kappa }_h{a}^2,$$

where the inequality uses Condition (C3). Note that since the derivative of log-partition function is the mean of the corresponding sufficient statistics and Ā_r(0; θ) = A(θ), $\frac{\partial }{\partial \eta }{\overline{A}}_r(0;{\theta }^{\ast })=\mathbb{E}[X_r^2]\le {\kappa }_v$ by assumption. Thus, by the standard Chernoff bounding technique, for all positive a ≤ 1,

$$P(\frac{1}{n}\sum _{i=1}^n{(X_r^{(i)})}^2\ge \delta )\le exp(-n\delta a+n{\kappa }_{v}a+\frac{n}{2}{\kappa }_h{a}^2).$$

With the choice of $a=\frac{\delta -{\kappa }_v}{\kappa h}\le 1$, we obtain

$$P(\frac{1}{n}\sum _{i=1}^n{(X_r^{(i)})}^2\ge \delta )\le exp(-n\frac{{(\delta -{\kappa }_v)}^2}{2{\kappa }_h})\le exp(-n\frac{{\delta }^2}{8{\kappa }_h}),$$

provided that δ ≤ 2κ_v/3, as in the statement.

Appendix C. Proof of Proposition 4

Let $\overline{v}\in {\mathbb{R}}^{p+(\begin{array}{c}p\\ 2\end{array})}$ be the zero-padded parameter with only one non-zero coordinate, which is 1, for the sufficient statistics X_r so that ‖v̄‖₂ = 1. A simple calculation shows that

$$log\mathbb{E}[exp(X_r)]=A({\theta }^{\ast }+\overline{v})-A({\theta }^{\ast }).$$

By a Taylor Series expansion and Condition (C3), we have for some ν ∈ [0, 1]

$$A({\theta }^{\ast }+\overline{v})-A({\theta }^{\ast })=\nabla A({\theta }^{\ast })\cdot \overline{v}+\frac{1}{2}{\overline{v}}^T{\nabla }^{2}A({\theta }^{\ast }+\nu \overline{v})\overline{v}\stackrel{(i)}{\le }\mathbb{E}[X_r]{\Vert \overline{v}\Vert }_2+\frac{1}{2}\frac{{\partial }^2}{\partial {\theta }_r^2}A({\theta }^{\ast }+\nu \overline{v}){\Vert \overline{v}\Vert }_2^2\le {\kappa }_m+\frac{1}{2}{\kappa }_h$$

where the inequality (i) uses the fact that v̄ has only nonzero element for the sufficient statistics X_r. Thus, again by the standard Chernoff bounding technique, for any positive constant a, $P(X_r\ge a)\le exp(-a+{\kappa }_m+\frac{1}{2}{\kappa }_h)$, and by setting a = δ log η we have

$$P(X_r\ge \delta log\eta )\le exp(-\delta log\eta +{\kappa }_m+\frac{1}{2}{\kappa }_h)\le c{\eta }^{-\delta }$$

where $c=exp({\kappa }_m+\frac{1}{2}{\kappa }_h)$, as claimed.

Appendix D. Proof of Theorem 5

In this section, we sketch the proof of Theorem 5 following the primal-dual witness proof technique in Wainwright (2009); Ravikumar et al. (2010). We first note that the optimality condition of the convex program (19) can be written as

$$\nabla \mathcal{l}(\widehat{\theta };X^{1:n})+{\lambda }_n\widehat{Z}=0$$ (27)

where Ẑ is a length p vector: Ẑ_\r ∈ ∂‖θ̂_\r‖₁ is a length (p – 1) subgradient vector where Ẑ_rt = sign(θ̂_rt) if θ̂_rt ≠ 0, and |Ẑ_rt| ≤ 1 otherwise; while Ẑ_r, corresponding to θ_r, is set to 0 since the nodewise term θ_r is not penalized in the M-estimation problem (19).

Note that in a high-dimensional regime with p ≫ n, the convex program (19) is not necessarily strictly convex, so that it might have multiple optimal solutions. However, the following lemma, adapted from Ravikumar et al. (2010), shows that nonetheless the solutions share their support set under certain conditions. We first recall the notation S = {(r, t) : t ∈ N*(r)} to denote the true neighborhood of node r, and S^c to denote its complement.

Lemma 8 Suppose that there exists a primal optimal solution θ̂ with associated subgradient Ẑ such that ‖Ẑ_S^c‖_∞ < 1. Then, any optimal solution θ̃ will satisfy θ̃_S^c = 0. Moreover, under the condition of ‖Ẑ_S^c‖_∞ < 1, if $Q_{SS}^{\ast }$ is invertible, then θ̂ is the unique optimal solution of (19).

Proof This lemma can be proved by the same reasoning developed for the special cases Wainwright (2009); Ravikumar et al. (2010) in our framework; As in the previous works, for any node-conditional distribution in the form of exponential family, we are solving the convex objective with ℓ₁ regularizer (19). Therefore, the problem can be written as an equivalent constrained optimization problem, and by the complementary slackness, for any optimal solution θ̃, we have 〈Ẑ, θ̃〉 = ‖θ̃‖₁. This simply implies that for all index j for which |Ẑ_j| < 1, θ̃_j = 0 (See Ravikumar et al. (2010) for details). Therefore, if there exists a primal optimal solution θ̂ with associated subgradient Ẑ such that ‖Ẑ_S^c‖_∞ < 1, then, any optimal solution θ̃ will satisfy θ̃_S^c = 0 as claimed.

Finally, we consider the restricted optimization problem subject to the constraint θ_S^C = 0. For this restricted optimization problem, if the Hessian, $Q_{SS}^{\ast }$, is positive definite as assumed in the lemma, then, this restricted problem is strictly convex, and its solution is unique. Moreover, since all primal optimal solutions of (19), θ̃, satisfy θ̃_S^c = 0 as discussed, the solution of the restricted problem is the unique solution of (19).

We use this lemma to prove the theorem following the primal-dual witness proof technique in Wainwright (2009); Ravikumar et al. (2010). Specifically, we explicitly construct a pair (θ̂, Ẑ) as follows (denoting the true support set of the edge parameters by S):

1. Recall that θ(r) = (θ_r, θ_\r) ∈ ℝ × ℝ^p−1. We first fix θ_S^c = 0 and solve the restricted optimization problem: (θ̂_r, θ̂_S, 0) = arg min_{θr∈ℝ, (θS,0)∈ℝ^p−1} {ℓ(θ; X^1:n)+λ_n‖θ_S‖₁}, and Ẑ_S = sign(θ̂_S).

2. We set θ̂_S^c = 0.

3. We set Ẑ_S^c to satisfy the condition (27) with θ̂ and Ẑ_S.

By construction, the support of θ̂ is included in the true support S of θ*, so that we would finish the proof of the theorem provided (a) θ̂ satisfies the stationary condition of (19), as well as the condition ‖Ẑ_S^c‖_∞ < 1 in Lemma 8 with high probability, so that by Lemma 8, the primal solution θ̂ is guaranteed to be unique; and (b) the support of θ̂ is not strictly within the true support S. We term these conditions strict dual feasibility, and sign consistency respectively.

We will now rewrite the subgradient optimality condition (27) as

$${\nabla }^2\mathcal{l}({\theta }^{\ast };X^{1:n})(\widehat{\theta }-{\theta }^{\ast })=-{\lambda }_n\widehat{Z}+W^n+R^n$$

where Wⁿ := −∇ℓ(θ*; X^1:n) is the sample score function (that we will show is small with high probability), and Rⁿ is the remainder term after coordinate-wise applications of the mean value theorem; $R_j^n={[{\nabla }^2\mathcal{l}({\theta }^{\ast };X^{1:n})-{\nabla }^2\mathcal{l}({\overline{\theta }}^{(j)};X^{1:n})]}_j^T(\widehat{\theta }-{\theta }^{\ast })$, for some θ̄^(j) on the line between θ̂ and θ*, and with ${[\cdot ]}_j^T$ being the j-th row of a matrix.

Recalling the notation for the Fisher information matrix Q* := ∇²ℓ(θ*; X^1:n), we then have

$$Q^{\ast }(\widehat{\theta }-{\theta }^{\ast })=-{\lambda }_n\widehat{Z}+W^n+R^n.$$

From now on, we provide lemmas that respectively control various terms in the above expression: the score term Wⁿ, the deviation θ̂ − θ*, and the remainder term Rⁿ. The first lemma controls the score term Wⁿ:

Lemma 9 Suppose that we set λ_n to satisfy $\frac{8(2-\alpha )}{\alpha }\sqrt{{\kappa }_1(n,p){\kappa }_4}\sqrt{\frac{logp}{n}}\le {\lambda }_n\le \frac{4(2-\alpha )}{\alpha }{\kappa }_1(n,p){\kappa }_2(n,p){\kappa }_4$ for some constant κ₄ ≤ min{2κ_v/3, 2κ_h + κ_v}. Suppose also that $n\ge \frac{8{\kappa }_h^2}{{\kappa }_4^2}logp$. Then, given a incoherence parameter α ∈ (0, 1],

$$P(\frac{2-\alpha }{{\lambda }_n}{\Vert W^n\Vert }_{\infty }\le \frac{\alpha }{4})\ge 1-c_1{p\prime }^{-2}-exp(-c_{2}n)-exp(-c_{3}n)$$

where p′ := max{n, p}.

The next lemma controls the deviation θ̂ – θ*.

Lemma 10 Suppose that ${\lambda }_{n}d\le \frac{{\rho }_{min}^2}{40{\rho }_{max}{\kappa }_3(n,p)logp\prime }$ and ${\Vert W^n\Vert }_{\infty }\le \frac{{\lambda }_n}{4}$. Then, we have

$$P({\Vert {\widehat{\theta }}_S-{\theta }_S^{\ast }\Vert }_2\le \frac{5}{{\rho }_{min}}\sqrt{d}{\lambda }_n)\ge 1-c_1{p\prime }^{-2},$$ (28)

for some constant c₁ > 0.

The last lemma controls the Taylor series remainder term Rⁿ:

Lemma 11 If ${\lambda }_{n}d\le \frac{{\rho }_{min}^2}{400{\rho }_{max}{\kappa }_3(n,p)logp\prime }\frac{\alpha }{2-\alpha }$, and ${\Vert W^n\Vert }_{\infty }\le \frac{{\lambda }_n}{4}$, then we have

$$P(\frac{{\Vert R^n\Vert }_{\infty }}{{\lambda }_n}\le \frac{\alpha }{4(2-\alpha )})\ge 1-c_1{p\prime }^{-2},$$ (29)

for some constant c₁ > 0.

The proof then follows from Lemmas 9, 10 and 11 in a straightforward fashion, following Ravikumar et al. (2010). Consider the choice of regularization parameter ${\lambda }_n=\frac{8(2-\alpha )}{\alpha }\sqrt{{\kappa }_1(n,p){\kappa }_4}\sqrt{\frac{logp}{n}}$. For a sample size greater $n\ge max\{\frac{4}{{\kappa }_1(n,p){\kappa }_2{(n,p)}^2{\kappa }_4},\frac{8{\kappa }_h^2}{{\kappa }_4^2}\}logp$, the conditions of Lemma 9 are satisfied, so that we may conclude that ${\Vert W^n\Vert }_{\infty }\le \frac{\alpha }{1-\alpha }\frac{{\lambda }_n}{4}\le \frac{{\lambda }_n}{4}$ with high probability. Moreover, with a sufficiently large sample size such that $n\ge L\prime {(\frac{2-\alpha }{\alpha })}^4{d}^2{\kappa }_1(n,p){\kappa }_3{(n,p)}^{2}logp{(logp\prime )}^2$ for some constant L′ > 0 depending only on ρ_min, ρ_max, κ₄ and α, it can be shown that the remaining condition of Lemma 11 (and hence the milder condition in Lemma 10) in turn is satisfied. Therefore, the resulting statements (28) and (29) of Lemmas 10 and 11 hold with high probability.

Strict dual feasibility

Following Ravikumar et al. (2010), we obtain

$${\Vert {\widehat{Z}}_{S^c}\Vert }_{\infty }\le {|\Vert Q_{S^{c}S}^{\ast }{(Q_{SS}^{\ast })}^{-1}\Vert |}_{\infty }[\frac{{\Vert W_S^n\Vert }_{\infty }}{{\lambda }_n}+\frac{{\Vert R_S^n\Vert }_{\infty }}{{\lambda }_n}+1]+\frac{{\Vert W_{S^c}^n\Vert }_{\infty }}{{\lambda }_n}+\frac{{\Vert R_{S^c}^n\Vert }_{\infty }}{{\lambda }_n}\le (1-\alpha )+(2-\alpha )[\frac{{\Vert W^n\Vert }_{\infty }}{{\lambda }_n}+\frac{{\Vert R^n\Vert }_{\infty }}{{\lambda }_n}]\le (1-\alpha )+\frac{\alpha }{4}+\frac{\alpha }{4}=1-\frac{\alpha }{2}<1.$$

Correct sign recovery

To guarantee that the support of θ̂ is not strictly within the true support S, it suffices to show that ${\Vert {\widehat{\theta }}_S-{\theta }_S^{\ast }\Vert }_{\infty }\le \frac{{\theta }_{min}^{\ast }}{2}$. From Lemma 10, we have ${\Vert {\widehat{\theta }}_S-{\theta }_S^{\ast }\Vert }_{\infty }\le {\Vert {\widehat{\theta }}_S-{\theta }_S^{\ast }\Vert }_2\le \frac{5}{{\rho }_{min}}\sqrt{d}{\lambda }_n\le \frac{{\theta }_{min}^{\ast }}{2}$ as long as ${\theta }_{min}^{\ast }\ge \frac{10}{{\rho }_{min}}\sqrt{d}{\lambda }_n$. This completes the proof.

D.1 Proof of Lemma 9

For a fixed t ∈ V\r, we define $V_t^{(i)}$ for notational convenience so that

$$W_t^n=\frac{1}{n}\sum _{i=1}^n{X}_r^{(i)}{X}_t^{(i)}-X_t^{(i)}D\prime ({\theta }_r^{\ast }+\langle {\theta }_{\backslash r}^{\ast },X_{V\backslash r}^{(i)}\rangle )=\frac{1}{n}\sum _{i=1}^n{V}_t^{(i)}$$

Consider the upper bound on the moment generating function of $V_t^{(i)}$, conditioned on $X_{V\backslash r}^{(i)}$,

$$\begin{array}{l}\mathbb{E}[exp(a{V}_t)|X_{V\backslash r}^{(i)}]={\int }_{X_r}exp\{a[X_r{X}_t^{(i)}-X_t^{(i)}D\prime ({\theta }_r^{\ast }+\langle {\theta }_{\backslash r}^{\ast },X_{V\backslash r}^{(i)}\rangle )]+(X_r({\theta }_r^{\ast }+\langle {\theta }_{\backslash r}^{\ast },X_{V\backslash r}^{(i)}\rangle )+C(X_r)-D({\theta }_r^{\ast }+\langle {\theta }_{\backslash r}^{\ast },X_{V\backslash r}^{(i)}\rangle ))\}\hfill \\ =exp\{D({\theta }_r^{\ast }+\langle {\theta }_{\backslash r}^{\ast },X_{V\backslash r}^{(i)}\rangle +a{X}_t^{(i)})-D({\theta }_r^{\ast }+\langle {\theta }_{\backslash r}^{\ast },X_{V\backslash r}^{(i)}\rangle )-a{X}_t^{(i)}D\prime ({\theta }_r^{\ast }+\langle {\theta }_{\backslash r}^{\ast },X_{V\backslash r}^{(i)}\rangle )\}\hfill \\ =exp\{\frac{a^2}{2}{(X_t^{(i)})}^{2}D″({\theta }_r^{\ast }+\langle {\theta }_{\backslash r}^{\ast },X_{V\backslash r}^{(i)}\rangle +{\nu }_{i}a{X}_t^{(i)})\}\text{for some}{\nu }_i\in [0,1]\hfill \end{array}$$

where the last equality holds by the second-order Taylor series expansion. Consequently, we have

$$\frac{1}{n}\sum _{i=1}^{n}log\mathbb{E}[exp(a{V}_t^{(i)})|X_{v\backslash r}^{(i)}]\le \frac{1}{n}\sum _{i=1}^n\frac{a}{2}{(X_t^{(i)})}^{2}D″({\theta }_r^{\ast }+\langle {\theta }_{\backslash r}^{\ast },X_{V\backslash r}^{(i)}\rangle +{\nu }_{i}a{X}_t^{(i)}).$$

First, we define the event: ${\xi }_1:=\{{max}_{i,s}|X_r^{(i)}|\le 4logp\prime \}$. Then, by Proposition 4, we obtain $P[{\xi }_1^c]\le c_{1}np{p\prime }^{-4}\le c_1{p\prime }^{-2}$. Provided that a ≤ κ₂(n, p), we can use Condition (C4) to control the second-order derivative of log-partition function and we obtain

$$\frac{1}{n}\sum _{i=1}^{n}log\mathbb{E}[exp(a{V}_t^{(i)})|X_{V\backslash r}^{(i)}]\le \frac{{\kappa }_1(n,p)a^2}{2}\frac{1}{n}\sum _{i=1}^n{(X_t^{(i)})}^2\text{for a}\le {\kappa }_2(n,p)$$

with probability at least 1 – c₁p′⁻². Now, for each index t, the variables $\frac{1}{n}{\sum }_{i=1}^n{(X_t^{(i)})}^2$ satisfy the tail bound in Proposition 3. Let us define the event ${\xi }_2:=\{\underset{t\in V}{max}\frac{1}{n}{\sum }_{i=1}^n{(X_t^{(i)})}^2\le {\kappa }_4\}$ for some constant κ₄ ≤ min{2κ_v/3, 2κ_h + κ_v}. Then, we can establish the upper bound of probability $P[{\xi }_2^c]$ by a union bound:

$$P[{\xi }_2^c]\le exp(-\frac{{\kappa }_4^2}{4{\kappa }_h^2}n+logp)\le exp(-c_{2}n)$$

as long as $n\ge \frac{8{\kappa }_h^2}{{\kappa }_4^2}logp$. Therefore, conditioned on ξ₁, ξ₂, the moment generating function is bounded as follows:

$$\frac{1}{n}\sum _{i=1}^{n}log\mathbb{E}[exp(a{V}_t^{(i)})|X_{V\backslash r}^{(i)},{\xi }_1,{\xi }_2]\le \frac{{\kappa }_1(n,p){\kappa }_4{a}^2}{2}\text{for a}\le {\kappa }_2(n,p).$$

The standard Chernoff bound technique implies that for any δ > 0,

$$P[|\frac{1}{n}\sum _{i=1}^n{V}_t^{(i)}|>\delta |{\xi }_1,{\xi }_2]\le 2exp(n(\frac{{\kappa }_1(n,p){\kappa }_4{a}^2}{2}-a\delta ))\text{for a}\le {\kappa }_2(n,p).$$

Setting $a=\frac{\delta }{{\kappa }_1(n,p){\kappa }_4}$ Yields

$$P[|\frac{1}{n}\sum _{i=1}^n{V}_t^{(i)}|>\delta |{\xi }_1,{\xi }_2]\le 2exp(-\frac{n{\delta }^2}{2{\kappa }_1(n,p){\kappa }_4})\text{for}\delta \le {\kappa }_1(n,p){\kappa }_2(n,p){\kappa }_4.$$

Suppose that $\frac{\alpha }{2-\alpha }\frac{{\lambda }_n}{4}\le {\kappa }_1(n,p){\kappa }_2(n,p){\kappa }_4$ for large enough n; thus setting $\delta =\frac{\alpha }{2-\alpha }\frac{{\lambda }_n}{4}$:

$$P[|\frac{1}{n}\sum _{i=1}^n{V}_t^{(i)}|>\frac{\alpha }{2-\alpha }\frac{{\lambda }_n}{4}|{\xi }_1,{\xi }_2]\le 2exp(-\frac{{\alpha }^2}{{(2-\alpha )}^2}\frac{n{\lambda }_n^2}{32{\kappa }_1(n,p){\kappa }_4}),$$

and by a union bound, we obtain

$$P[{\Vert W^n\Vert }_{\infty }>\frac{\alpha }{2-\alpha }\frac{{\lambda }_n}{4}|{\xi }_1,{\xi }_2]\le 2exp(-\frac{{\alpha }^2}{{(2-\alpha )}^2}\frac{n{\lambda }_n^2}{32{\kappa }_1(n,p){\kappa }_4}+logp).$$

Finally, provided that ${\lambda }_n\ge \frac{8(2-\alpha )}{\alpha }\sqrt{{\kappa }_1(n,p){\kappa }_4}\sqrt{\frac{logp}{n}}$, we obtain

$$P[{\Vert W^n\Vert }_{\infty }>\frac{\alpha }{2-\alpha }\frac{{\lambda }_n}{4}]\le c_1{p\prime }^{-2}+exp(-c_{2}n)+exp(-c_{3}n)$$

where we use the fact that the probability of occurring event 𝒜 is upper bounded by $P(\mathcal{A})\le P({\xi }_1^c)+P({\xi }_2^c)+P(\mathcal{A}|{\xi }_1,{\xi }_2)$.

D.2 Proof of Lemma 10

In order to establish the error bound ${\Vert {\widehat{\theta }}_S-{\theta }_S^{\ast }\Vert }_2\le B$ for some radius B, several works (e.g. Negahban et al. (2012); Ravikumar et al. (2010)) proved that it suffices to show F(u_S) > 0 for all $u_S:={\theta }_S-{\theta }_S^{\ast }$ s.t. ‖u_S‖₂ = B where

$$F(u_S):=\mathcal{l}({\theta }_S^{\ast }+u_S;X^{1:n})-\mathcal{l}({\theta }_S^{\ast };X^{1:n})+{\lambda }_n({\Vert {\theta }_S^{\ast }+u_S\Vert }_1-{\Vert {\theta }_S^{\ast }\Vert }_1).$$

Note that F(0) = 0, and for ${\widehat{u}}_S:={\widehat{\theta }}_S-{\theta }_S^{\ast }$, F(u_S) ≤ 0. From now on, we show that F(u_S) is strictly positive on the boundary of the ball with radius B = Mλ_n√d where M > 0 is a parameter that we will choose later in this proof. Some algebra yields

$$F(u_S)\ge {({\lambda }_n\sqrt{d})}^2\{-\frac{1}{4}M+q^{\ast }{M}^2-M\}$$ (30)

where q* is the minimum eigenvalue of ${\nabla }^2\mathcal{l}({\theta }_S^{\ast }+v{u}_S;X^{1:n})$ for some v ∈ [0, 1]. Moreover,

$$\begin{array}{l}{q}^{\ast }:={\mathrm{\Lambda }}_{min}({\nabla }^2\mathcal{l}({\theta }_S^{\ast }+v{u}_S))\hfill \\ \ge \underset{v\in [0,1]}{min}{\mathrm{\Lambda }}_{min}({\nabla }^2\mathcal{l}({\theta }_S^{\ast }+v{u}_S))\hfill \\ \ge {\mathrm{\Lambda }}_{min}[\frac{1}{n}\sum _{i=1}^{n}D″({\theta }_r^{\ast }+\langle {\theta }_S^{\ast },X_S^{(i)}\rangle )X_S^{(i)}{(X_S^{(i)})}^T]-\underset{v\in [0,1]}{max}{|\Vert \frac{1}{n}\sum _{i=1}^{n}D‴({\theta }_r^{\ast }+\langle {\theta }_S^{\ast },X_S^{(i)}\rangle )(u_S^T{X}_S^{(i)})X_S^{(i)}{(X_S^{(i)})}^T\Vert |}_2\hfill \\ \ge {\rho }_{min}-\underset{v\in [0,1]}{max}\underset{y}{max}\frac{1}{n}\sum _{i=1}^n|D‴({\theta }_r^{\ast }+\langle {\theta }_S^{\ast },X_S^{(i)}\rangle )||\langle u_S,X_S^{(i)}\rangle |{(\langle X_S^{(i)},y\rangle )}^2\end{array}$$

where y ∈ ℝ^d s.t ‖y‖₂ = 1. Similarly as in the previous proof, we consider the event ξ₁ with probability at least 1 – c₁p′⁻². Then, since all the elements in vector $X_S^{(i)}$ is smaller than 4log p′, $|\langle u_S,X_S^{(i)}\rangle |\le 4logp\prime \sqrt{d}{\Vert u_S\Vert }_2=4logp\prime M{\lambda }_{n}d$ for all i. At the same time, by Condition (C4), $|D‴(({\theta }_r^{\ast }+v{u}_S)+\langle {\theta }_S^{\ast }+v{u}_S,X_S^{(i)}\rangle )|\le {\kappa }_3(n,p)$. Note that ${\theta }_S^{\ast }+v{u}_S$ is a convex combination of ${\theta }_S^{\ast }$ and θ̂_S, and as a result, we can directly apply the Condition (C4). Hence, conditioned on ξ₁, we have

$$q^{\ast }\ge {\rho }_{min}-4{\rho }_{max}M{\lambda }_{n}d{\kappa }_3(n,p)logp\prime .$$

As a result, assuming that ${\lambda }_n\le \frac{{\rho }_{min}}{8{\rho }_{max}Md{\kappa }_3(n,p)logp\prime },q^{\ast }\ge \frac{{\rho }_{min}}{2}$. Finally, from (30), we obtain

$$F(u_S)\ge {({\lambda }_n\sqrt{d})}^2\{-\frac{1}{4}M+\frac{{\rho }_{min}}{2}{M}^2-M\},$$

which is strictly positive for $M=\frac{5}{{\rho }_{min}}$. Therefore, if ${\lambda }_{n}d\le \frac{{\rho }_{min}}{8{\rho }_{max}M{\kappa }_3(n,p)logp\prime }\le \frac{{\rho }_{min}^2}{40{\rho }_{max}{\kappa }_3(n,p)logp\prime }$, then

$${\Vert {\widehat{\theta }}_S-{\theta }_S^{\ast }\Vert }_2\le \frac{5}{{\rho }_{min}}\sqrt{d}{\lambda }_n,$$

which completes the proof.

D.3 Proof of Lemma 11

In the proof, we are going to show that ${\Vert R^n\Vert }_{\infty }\le 4{\kappa }_3(n,p)logp\prime {\rho }_{max}{\Vert {\widehat{\theta }}_S-{\theta }_S^{\ast }\Vert }_2^2$. Then, since the conditions of Lemma 11 are stronger than those of Lemma 10, from the result of Lemma 10, we can conclude that

$${\Vert R^n\Vert }_{\infty }\le \frac{100{\kappa }_3(n,p){\rho }_{max}logp\prime }{{\rho }_{min}^2}{\lambda }_n^{2}d,$$

as claimed in Lemma 11.

From the definition of Rⁿ, for a fixed t ∈ V\r, $R_t^n$ can be written as

$$\frac{1}{n}\sum _{i=1}^n[D″({\theta }_r^{\ast }+\langle {\theta }_{\backslash r}^{\ast },X_{V\backslash r}^{(i)}\rangle )-D″({\overline{\theta }}_s+\langle {\overline{\theta }}^{(t)},X_{V\backslash r}^{(i)}\rangle )]{\left[X_{V\backslash r}^{(i)}{(X_{V\backslash r}^{(i)})}^T\right]}_t^T[{\widehat{\theta }}_{\backslash r}-{\theta }_{\backslash r}^{\ast }]$$

where ${\overline{\theta }}_{\backslash r}^{(t)}$ is some point in the line between θ̂_\r and ${\theta }_{\backslash r}^{\ast }$, i.e., ${\overline{\theta }}_{\backslash r}^{(t)}=v_t{\widehat{\theta }}_{\backslash r}+(1-v_t){\theta }_{\backslash r}^{\ast }$ for v_t ∈ [0, 1]. By another application of the mean value theorem, we have

$$R_t^n=-\frac{1}{n}\sum _{i=1}^n\{D‴({\stackrel{=}{\theta }}_s+\langle {\stackrel{=}{\theta }}_{\backslash r}^{(t)},X_{V\backslash r}^{(i)}\rangle )X_t^{(i)}\}\{v_t{[{\widehat{\theta }}_{\backslash r}-{\theta }_{\backslash r}^{\ast }]}^T{X}_{V\backslash r}^{(i)}{(X_{V\backslash r}^{(i)})}^T[{\widehat{\theta }}_{\backslash r}-{\theta }_{\backslash r}^{\ast }]\}$$

for a some point ${\stackrel{=}{\theta }}_{\backslash r}^{(t)}$ between ${\overline{\theta }}_{\backslash r}^{(t)}$ and ${\theta }_{\backslash r}^{\ast }$. Similarly in the previous proofs, conditioned on the event ξ₁, we obtain

$$|R_t^n|\le \frac{4{\kappa }_3(n,p)logp\prime }{n}\sum _{i=1}^n\{v_t{[{\widehat{\theta }}_{\backslash r}-{\theta }_{\backslash r}^{\ast }]}^T{X}_{V\backslash r}^{(i)}{(X_{V\backslash r}^{(i)})}^T[{\widehat{\theta }}_{\backslash r}-{\theta }_{\backslash r}^{\ast }]\}.$$

Performing some algebra yields

$$|R_t^n|\le 4{\kappa }_3(n,p){\rho }_{max}logp\prime {\Vert {\widehat{\theta }}_S-{\theta }_S^{\ast }\Vert }_2^2,\text{for all}t\in V\backslash r$$

with probability at least 1 – c₁p′⁻², which completes the proof.

Appendix E. Optimization Problems for Poisson and Exponential Graphical Model Neighborhood Selection

We propose to fit our family of graphical models by performing neighborhood selection, or maximizing the ℓ₁-penalized log-likelihood for each node conditional on all other nodes. For several exponential families, however, further restrictions on the parameter space are needed to ensure a proper Markov Random Field. When performing neighborhood selection, these can be imposed by adding constraints to the penalized generalized linear models. We illustrate this by providing the optimization problems solved by our Poisson graphical model and exponential graphical model M-estimator that are used in Section 4.

Following from Section 3, the neighborhood selection problem for our family of models maximizes the likelihood of a node, X_r, conditional on all other nodes, X_V\r. This conditional likelihood is regularized with an ℓ₁ penalty to induce sparsity in the edge weights, θ(r) a p – 1 dimensional vector, and constrained to enforce restrictions, θ(r) ∈ 𝒞, needed to yield a proper MRF:

$$\underset{\theta (r)}{\text{maximize}}\frac{1}{n}\sum _{i=1}^n\mathcal{l}(X_{i,r}|X_{i,V\backslash r};\theta (r))-{\lambda }_n{\Vert \theta (r)\Vert }_1\text{subject to}\theta (r)\in \mathcal{C},$$

where ℓ (X_i,r|X_i,V\r; θ(r)) is the conditional log-likelihood for the exponential family. For the Poisson graphical model, the edge weights are constrained to be non-positive. This yields the following optimization problem:

$$\begin{array}{l}\underset{\theta (r)}{\text{maximize}}\frac{1}{n}\sum _{i=1}^n[X_{r,i}{X}_{V\backslash r,i}^T\theta (r)-exp(X_{V\backslash r,i}^T\theta (r)\left)\right]-{\lambda }_n{\Vert \theta (r)\Vert }_1\hfill \\ \text{subject to}\theta (r)\le 0.\hfill \end{array}$$

Similarly, the edge weights of the exponential graphical are restricted to be non-negative yielding

$$\begin{array}{l}\underset{\theta (r)}{\text{maximize}}\frac{1}{n}\sum _{i=1}^n[-X_{r,i}{X}_{V\backslash r,i}^T\theta (r)+log(X_{V\backslash r,i}^T\theta (r)\left)\right]-{\lambda }_n{\Vert \theta (r)\Vert }_1\hfill \\ \text{subject to}\theta (r)\ge 0.\hfill \end{array}$$

Note that we neglect the intercept term, assuming this to be zero as is common in other proposed neighborhood selection methods (Meinshausen and Bühlmann, 2006; Ravikumar et al., 2010; Jalali et al., 2011). Both of the neighborhood selection problems are concave problems with a smooth log-likelihood and linear constraints. While there are a plethora of optimization routines available to solve such problems, we have employed a projected gradient descent scheme which is guaranteed to converge to a global optimum (Daubechies et al., 2008; Beck and Teboulle, 2010).