The Expxorcist: Nonparametric Graphical Models Via Conditional Exponential Densities

Abstract

Non-parametric multivariate density estimation faces strong statistical and computational bottlenecks, and the more practical approaches impose near-parametric assumptions on the form of the density functions. In this paper, we leverage recent developments to propose a class of non-parametric models which have very attractive computational and statistical properties. Our approach relies on the simple function space assumption that the conditional distribution of each variable conditioned on the other variables has a non-parametric exponential family form.

1. Introduction

Let X = (X₁, …, X_p) be a p-dimensional random vector. Let G = (V, E) be the graph that encodes conditional independence assumptions underlying the distribution of X, that is, each node of the graph corresponds to a component of vector X and (a, b) ∈ E if and only if $X_a⫫̸X_b\mid X_{\mathit{\neg ab}}$ with X_¬ab := {X_c ∣ c ∈ V \{a, b}}. The graphical model represented by G is then the set of distributions over X that satisfy the conditional independence assumptions specified by the graph G.

There has been a considerable line of work on learning parametric families of such graphical model distributions from data [22, 20, 13, 28], where the distribution is indexed by a finite-dimensional parameter vector. The goal of this paper, however, is on specifying and learning nonparametric families of graphical model distributions, indexed by infinite-dimensional parameters, and for which there has been comparatively limited work. Non-parametric multivariate density estimation broadly, even without the graphical model constraint, has not proved as popular in practical machine learning contexts, for both statistical and computational reasons. Loosely, estimating a non-parametric multivariate density, with mild assumptions, typically requires the number of samples to scale exponentially in the dimension p of the data, which is infeasible even in the big-data era when n is very large. And the resulting estimators are typically computationally expensive or intractable, for instance requiring repeated computations of multivariate integrals.

We present a review of multivariate density estimation, that is necessarily incomplete but sets up our proposed approach. A common approach dating back to [15] uses the logistic density transform to satisfy the unity and positivity constraints for densities, and considers densities of the form $f(X)=\frac{\text{exp}(\eta (X))}{{\int }_{\mathcal{X}}\text{exp}(\eta (x))\mathit{dx}}$, with some constraints on η for identifiability such as η(X₀) = 0 for some $X_0\in \mathcal{X}$ or ${\int }_{\mathcal{X}}\eta (x)\mathit{dx}=0$.

With the logistic density transform, differing approaches for non-parametric density estimation can be contrasted in part by their assumptions on the infinite-dimensional function space domain of η(·). An early approach [8] considered function spaces of functions with bounded “roughness” functionals. The predominant line of work however has focused on the setting where η(·) lies in a Reproducing Kernel Hilbert Space (RKHS), dating back to [21]. Consider the estimation of these logistic density transforms η(X) given n i.i.d. samples ${\mathbb{X}}_n={\left\{X^{(i)}\right\}}_{i=1}^n$ drawn from f_η(X). A natural loss functional is penalized log likelihood, with a penalty functional that ensures a smooth fit with respect to the function space domain: $\ell (\eta ;{\mathbb{X}}_n)≔-\frac{1}{n}{\sum }_{i\in \left[n\right]}\eta (X^{(i)})+\log \int \text{exp}(\eta (x))\mathit{dx}+\lambda \text{pen}(\eta )$, for functions η(·) that lie in an RKHS $\mathcal{H}$, and where $\text{pen}(\eta )={\Vert \eta \Vert }_{\mathcal{H}}^2$ is the squared RKHS norm. This was studied by many [21, 11, 6]. A crucial caveat is that the representer theorem for RKHSs does not hold. Nonetheless, one can consider finite-dimensional function space approximations consisting of the linear span of kernel functions evaluated at the sample points [12]. Computationally this still scales poorly with the dimension due to the need to compute multidimensional integrals of the form ∫ exp(η(x)dx which do not, in general, decompose. These approximations also do not come with strong statistical guarantees.

We briefly note that the function space assumption that η(·) lies in an RKHS could also be viewed from the lens of an infinite-dimensional exponential family [4]. Specifically, let $\mathcal{H}$ be a Reproducing Kernel Hilbert Space with reproducing kernel k(·, ·), and inner product ${\langle \cdot ,\cdot \rangle }_{\mathcal{H}}$. Then $\eta (X)={\langle \theta (\cdot ),k(X,\cdot )\rangle }_{\mathcal{H}}$, so that the density f (X) can in turn be viewed as a member of an infinite-dimensional exponential family with sufficient statistics $k(X,\cdot ):\mathcal{X}\mapsto \mathcal{H}$, and natural parameter $\theta (\cdot )\in \mathcal{H}$. Following this viewpoint, [4] propose estimators via linear span approximations similar to [11].

Due to the computational caveat with exact likelihood based functionals, a line of approaches have focused on penalized surrogate likelihoods instead. [14] study the following loss functional: $\ell (\eta ;{\mathbb{X}}_n)≔\frac{1}{n}{\sum }_{i\in \left[n\right]}\text{exp}(-\eta (X^{(i)}))+\int \eta (x)\rho (x)\mathit{dx}+\lambda \text{pen}(\eta )$, where ρ(X) is some fixed known density with the same support as the unknown density f (X). While this estimation procedure is much more computationally amenable than minimizing the exact penalized likelihood, the caveat, however, is that for a general RKHS this requires solving higher order integrals. The next level of simplification has thus focused on the form of the logistic transform function itself. There has been a line of work on an ANOVA type decomposition of the logistic density function into node-wise and pairwise terms: $\eta (X)={\sum }_{s=1}^p{\eta }_s(X_s)+{\sum }_{s=1}^p{\sum }_{t=s+1}^p{\eta }_{\mathit{st}}(X_s,X_t)$. A line of work has coupled such a decomposition with the assumption that each of the terms lie in an RKHS. This does not immediately provide a computational benefit: with penalized likelihood based loss functionals, the loss functional does not necessarily decompose into such node and pairwise terms. [24] thus couple this ANOVA type pairwise decomposition with a score matching based objective. [10] use the above decomposition with the surrogate loss functional of [14] discussed above, but note that this still requires the aforementioned function space approximation as a linear span of kernel evaluations, as well as two-dimensional integrals.

A line of recent work has thus focused on further stringent assumptions on the density function space, by assuming some components of the logistic transform to be finite-dimensional. [30] use an ANOVA decomposition but assume the terms belong to finite-dimensional function spaces instead of RKHSs, specified by a pre-defined finite set of basis functions. [29] consider logistic transform functions η(·) that have the pairwise decomposition above, with a specific class of parametric pairwise functions β_stX_sX_t, and non-parametric node-wise functions. [17, 16] consider the problem of estimating monotonic node-wise functions such that the transformed random vector is multivariate Gaussian; which could also be viewed as estimating a Gaussian copula density.

To summarize the (necessarily incomplete) review above, non-parametric density estimation faces strong statistical and computational bottlenecks, and the more practical approaches impose stringent near-parametric assumptions on the form of the (logistic transform of the) density functions. In this paper, we leverage recent developments to propose a very computationally simple non-parametric density estimation algorithm, that still comes with strong statistical guarantees. Moreover, the density could be viewed as a graphical model distribution, with a corresponding sparse conditional independence graph.

Our approach relies on the following simple function space assumption: that the conditional distribution of each variable conditioned on the other variables has a non-parametric exponential family form. As we show, for there to exist a consistent joint density, the logistic density transform with respect to a particular base measure necessarily decomposes into the following semi-parametric form: $\eta (X)={\sum }_{s=1}^p{\theta }_s{B}_s(X_s)+{\sum }_{s=1}^p{\sum }_{t=s+1}^p{\theta }_{\mathit{st}}{B}_s(X_s)B_t(X_t)$ in the pairwise case, with both a parametric component {θ_s : s = 1, …, p}, {θ_st : s < t; s, t = 1, …, p}, as well as non-parametric components {B_s : s = 1, …, p}. We call this class of models the “expxorcist”, following other “ghostbusting” semi-parametric models such as the nonparanormal and nonparanormal skeptic [17, 16].

Since the conditional distributions are exponential families, we show that there exist computationally amenable estimators, even in our more general non-parametric setting, where the sufficient statistics have to be estimated as well. The statistical analysis in our non-parametric setting however is more subtle, due in part to non-convexity and in part to the non-parametric setting. We also show how the Expxorcist class of densities is closely related to a semi-parametric exponential family copula density that generalizes the Gaussian copula density of [17, 16]. We corroborate the applicability of our class of models with experiments on synthetic and real data sets.

2. Multivariate Density Specification via Conditional Densities

We are interested in the approach of estimating a multivariate density by estimating node-conditional densities. Since node-conditional densities focus on the density of a single variable, though conditioned on the rest of the variables, estimating these is potentially a simpler problem, both statistically and computationally, than estimating the entire joint density itself. Let us consider the general non-parametric conditional density estimation problem. Given the general multivariate density $f(X)=\frac{\text{exp}(\eta (X))}{{\int }_{\mathcal{X}}\text{exp}(\eta (x))\mathit{dx}}$, the conditional density of a variable X_s given the rest of the variables X_−s is given by $f(X_s\mid X_{-s})=\frac{\text{exp}(\eta ((X_s,X_{-s})))}{{\int }_{{\mathcal{X}}_s}\text{exp}(\eta ((x,X_{-s})))\mathit{dx}}$, which does not have a multi-dimensional integral, but otherwise does not have a computationally amenable form. There has been a line of work on such conditional density estimation, mirroring developments in multivariate density estimation [9, 18, 23], but unlike parametric settings, there are no large sample complexity gains with non-parametric conditional density estimation under general settings. There have also been efforts to use ANOVA decompositions in a conditional density context [31, 26].

In addition to computational and sample complexity caveats, recall that in our context, we would like to use conditional density estimates to infer a joint multivariate density. A crucial caveat with using the above estimates to do so is that it is not clear when the estimated node-conditional densities would be consistent with a joint multivariate density. There has been a line of work on this question (of when conditional densities are consistent with a joint density) for parametric densities; see [1] for an overview, with more recent results in [27, 5, 2, 25]. Overall, while estimating node-conditional densities could be viewed as surrogate estimation of a joint density, arbitrary node-conditional distributions need not be consistent in general with any joint density. There has however been a line of work in recent years [3, 28], where it was shown that when the node-conditional distributions belong to an exponential family, then under certain conditions on their parameterization, there do exist multivariate densities consistent with the node-conditional densities. In the next section, we leverage these results towards non-parametric estimation of conditional densities.

3. Conditional Densities of an Exponential Family Form

We first recall the definition of an exponential family in the context of a conditional density.

Definition 1. A conditional density of a random variable $Y\in \mathcal{Y}$ given covariates $Z≔(Z_1,\dots ,Z_m)\in \mathcal{Z}$ is said to have an exponential family form if it can be written as f(Y ∣ Z) = exp(B(Y)^T E(Z) + C(Y) + D(Z)), for some functions $B:\mathcal{Y}\mapsto {\mathbb{R}}^k$ (for some finite integer k > 0), $E:\mathcal{Z}\mapsto {\mathbb{R}}^k,C:\mathcal{Y}\mapsto \mathbb{R}$ and $D:\mathcal{Z}\mapsto \mathbb{R}$.

Thus, f (Y ∣ Z) belongs to a finite-dimensional exponential family with sufficient statistics B(Y), base measure exp(C(Y)), and with natural parameter E(Z) and where −D(Z) is the log-partition function. Contrast this with a general conditional density f (Y ∣ Z) = exp(h(Y, Z) + C(Y) + D(Z)) with respect to the base measure exp(C(Y )) and −D(Z) being the log-normalization constant, and it can be seen that a conditional density of the exponential family form has its logistic density transform h(Y, Z) that factorizes as B(Y)^T E(Z).

Consider the case where the sufficient statistic function is real-valued. The non-parametric estimation problem of a conditional density of exponential form then reduces to the estimation of the sufficient statistics function B(·), the exponential natural parameter function E(·), assuming the base measure C(·) is given. But when would such estimated conditional densities be consistent with a joint density? To answer this question, we draw upon developments in [28]. Suppose that the node-conditional distributions of each random variable X_s conditioned on the rest of random variables have the exponential family form as in Definition 1, so that for each s ∈ V

$$\mathbb{P}(X_s\mid X_{-s})\propto \text{exp}\{E_s(X_{-s})B_s(X_s)+C_s(X_s)\},$$ (1)

for some arbitrary functions E_s(·), B_s(·), C_s(·) that specify a valid conditional density. Then [28] show that these node-conditional densities are consistent with a unique joint density over the random vector X, that moreover factors according to a set of cliques $\mathcal{C}$ in the graph G, if and only if the functions {E_s(·)}_s∈V specifying the node-conditional distributions have the form $E_s(X_{-s})={\theta }_s+{\sum }_{C\in \mathcal{C}:s\in C}{\theta }_C{\prod }_{t\in C,t\ne s}{B}_t(X_t)$, where $\left\{{\theta }_s\right\}\cup {\left\{{\theta }_C\right\}}_{C\in \mathcal{C}}$ is a set of parameters. Moreover, the corresponding consistent joint distribution has the following form

$$\mathbb{P}(X)\propto \text{exp}\left\{{\sum }_{s\in V}{\theta }_s{B}_s(X_s)+{\sum }_{C\in \mathcal{C}}{\theta }_C{\prod }_{s\in C}{B}_s(X_s)+{\sum }_{s\in V}{C}_s(X_s)\right\}.$$ (2)

In this paper, we are interested in the non-parametric estimation of the Expxorcist class of densities in (2), where we estimate both the finite-dimensional parameters $\left\{{\theta }_s\right\}\cup {\left\{{\theta }_C\right\}}_{C\in \mathcal{C}}$, as well as the functions {B_s(X_s)}_s∈V. We assume we are given the base measures {C_s(X_s)}_s∈V, so that the joint density is with respect to a given product base measure ∏_s∈V exp(C_s(X_S)), as is common in the multivariate density estimation literature. Note that this is not a very restrictive assumption. In practice the base measure at each node can be well approximated using the empirical univariate marginal density of that node. We could also extend our algorithm, which we present next, to estimate the base measures along with sufficient statistic functions.

4. Regularized Conditional Likelihood Estimation for Exponential Family Form Densities

We consider the nonparametric estimation problem of estimating a joint density of the form in (2), focusing on the pairwise case where the factors have size at most k = 2, so that the joint density takes the form

$$\mathbb{P}(X)\propto \text{exp}\left\{{\sum }_{s\in V}{\theta }_s{B}_s(X_s)+{\sum }_{(s,t)\in E}{\theta }_{\mathit{st}}{B}_s(X_s)B_t(X_t)+{\sum }_{s\in V}{C}_s(X_s)\right\}.$$ (3)

As detailed in the previous section, estimating this joint density can be reduced to estimating its node-conditional densities, which take the form

$$\mathbb{P}(X_s\mid X_{-s})\propto \text{exp}\left\{B_s(X_s)\left({\theta }_s+{\sum }_{t\in N_G(s)}{\theta }_{\mathit{st}}{B}_t(X_t)\right)+C_s(X_s)\right\}.$$ (4)

We now introduce some notation which we use in the sequel. Let Θ = {θ_s}_s∈V ∪ {θ_st}_s≠t and Θ_s = θ_s ∪ {θ_st}_t∈V\{s}. Let B = {B_s}_s∈V be the set of sufficient statistics. Let ${\mathcal{X}}_s$ be the domain of X_s, which we assume is bounded and $L^2({\mathcal{X}}_s)$ be the Hilbert space of square integrable functions over ${\mathcal{X}}_s$ with respect to Lebesgue measure. We assume that the sufficient statistics $B_s(\cdot )\in L^2({\mathcal{X}}_s)$.

Note that the model in Equation (3) is unidentifiable. To overcome this issue we impose additional constraints on its parameters. Specifically, we require B_s(X_s) to satisfy ${\int }_{{\mathcal{X}}_s}{B}_s(X)\mathit{dX}=0$, ${\int }_{{\mathcal{X}}_s}{B}_s{(X)}^2\mathit{dX}=1$ and θ_s ≥ 0, ∀s ∈ V.

Optimization objective:

Let ${\mathbb{X}}_n=\{X^{(1)},\dots X^{(n)}\}$ be n i.i.d. samples drawn from a joint density of the form in Equation (3), with parameters Θ*, B*. And let ${\mathcal{L}}_s({\mathrm{\Theta }}_s,B;{\mathbb{X}}_n)$ be the node conditional negative log likelihood at node s

$${\mathcal{L}}_s({\mathrm{\Theta }}_s,B;{\mathbb{X}}_n)=\frac{1}{n}{\sum }_{i=1}^n\left\{-B_s(X_s^{(i)})\left({\theta }_s+{\sum }_{t\in V\backslash s}{\theta }_{\mathit{st}}{B}_t(X_t^{(i)})\right)+A(X_{-s}^{(i)};{\mathrm{\Theta }}_s,B)\right\},$$

where A(X_−s; Θ_s, B) is the log partition function. To estimate the unknown parameters, we solve the following regularized node conditional log-likelihood estimation problem at each node s ∈ V

$$\begin{array}{c}\hfill \underset{{\mathrm{\Theta }}_s,B}{\text{min}}{\mathcal{L}}_s({\mathrm{\Theta }}_s,B;{\mathbb{X}}_n)+{\lambda }_n{\Vert {\mathrm{\Theta }}_s\Vert }_1\hfill \\ \hfill \text{s.t.}{\theta }_s\ge 0,{\int }_{{\mathcal{X}}_i}{B}_t(X)\mathit{dX}=0{\int }_{{\mathcal{X}}_t}{B}_t{(X)}^2\mathit{dX}=1\forall t\in V.\hfill \end{array}$$ (5)

The equality constraints on the norm of functions B_t(·) makes the above optimization problem a difficult one to solve. While the norm constraints on B_t(·), ∀t ∈ V \ s can be handled through re-parametrization, the constraint on B_s(·) can not be handled efficiently. To make the optimization more amenable for numerical optimization techniques, we solve a closely related optimization problem. At each node s ∈ V, we consider the following re-parametrization of B: B_s(X_s) ← θ_sB_s(X_s), B_t(X_t) ← (θ_st/θ_s)B_t(X_t), ∀t ∈ V \ {s}. With a slight abuse of notation we redefine ${\mathcal{L}}_s$ using this re-parametrization as

$${\mathcal{L}}_s(B;{\mathbb{X}}_n)=\frac{1}{n}{\sum }_{i=1}^n\left\{-B_s(X_s^{(i)})\left(1+{\sum }_{t\in V\backslash s}{B}_t(X_t^{(i)})\right)+A(X_{-s}^{(i)};B)\right\},$$ (6)

where A(X_−s; B) is the log partition function. We solve the following optimization problem, which is closely related to the original optimization in Equation (5)

$$\begin{array}{c}\hfill \underset{B}{\text{min}}{\mathcal{L}}_s(B;{\mathbb{X}}_n)+{\lambda }_n\sum _{t\in V}\sqrt{{\int }_{{\mathcal{X}}_t}{B}_t{(X)}^2\mathit{dX}}\hfill \\ \hfill \text{s.t.}{\int }_{{\mathcal{X}}_t}{B}_t(X)\mathit{dX}=0\forall t\in V.\hfill \end{array}$$ (7)

For more details on the relation between (5) and (7), please refer to Appendix.

Algorithm:

We now present our algorithm for optimization of (7). In the sequel, for simplicity, we assume that the domains ${\mathcal{X}}_t$ of random variables X_t are all the same and equal to $\mathcal{X}$. In order to estimate functions B_t, we expand them over a uniformly bounded, orthonormal basis {${\{{\varphi }_k(\cdot )\}}_{k=0}^{\infty }$ of $L^2(\mathcal{X})$ with ϕ₀(·) ∝ 1. Expansion of the functions B_t(·) over this basis yields

$$B_t(X)={\sum }_{k=1}^m{\alpha }_{t,k}{\varphi }_k(X)+{\rho }_{t,m}(X)\text{where}{\rho }_{t,m}(X)={\alpha }_{t,0}{\varphi }_0(X)+{\sum }_{k=m+1}^{\infty }{\alpha }_{t,k}{\varphi }_k(X).$$

Note that the constraint ${\int }_{\mathcal{X}}{B}_t(X)\mathit{dX}=0$ in Equation (7), translates to α_t,0 = 0. To convert the infinite dimensional optimization problem in (7) into a finite dimensional problem, we truncate the basis expansion to the top m terms and approximate B_t(·) as ${\sum }_{k=1}^m{\alpha }_{t,k}{\varphi }_k(\cdot )$. The optimization problem in Equation (7) can then be rewritten as

$$\underset{{\alpha }_{\mathrm{m}}}{\text{min}}{\mathcal{L}}_{s,m}({\alpha }_{\mathrm{m}};{\mathbb{X}}_n)+{\lambda }_n\sum _{t\in V}{\Vert {\alpha }_{\mathrm{t},\mathrm{m}}\Vert }_2,$$ (8)

where ${\alpha }_{\mathrm{t},\mathrm{m}}={\left\{{\alpha }_{t,k}\right\}}_{k=1}^m$, α_m = {α_t,m}_t∈V and ${\mathcal{L}}_{s,m}$ is defined as

$${\mathcal{L}}_{s,m}({\alpha }_{\mathrm{m}};{\mathbb{X}}_n)=\frac{1}{n}\sum _{i=1}^n\left\{-\sum _{k=1}^m{\alpha }_{s,k}{\varphi }_k(X_s^{(i)})\left(1+\sum _{t\in V\backslash \left\{s\right\}}\sum _{l=1}^m{\alpha }_{t,l}{\varphi }_l(X_t^{(i)})\right)+A(X_{-s}^{(i)};{\alpha }_{\mathrm{m}})\right\}.$$

Iterative minimization of (8):

Note that the objective in (8) is non-convex. In this work, we use a simple alternating minimization technique for its optimization. In this technique, we alternately minimize α_s,m, {α_t,m}_t∈V\s while fixing the other parameters. The resulting optimization problem in each of the alternating steps is convex. We use Proximal Gradient Descent to optimize these sub-problems. To compute the objective and its gradients, we need to numerically evaluate the one-dimensional integrals in the log partition function. To do this, we choose a uniform grid of points over the domain and use quadrature rules to approximate the integrals.

Convergence:

Although (8) is non-convex, we can show that under certain conditions on the objective function, the alternating minimization procedure converges to the global minimum. In a recent work [32] analyze alternating minimization for low rank matrix factorization problems and show that it converges to a global minimum if the sequence of convex problems are strongly convex and satisfy certain other regularity condition. The analysis of [32] can be extended to show global convergence of alternating minimization for (8).

5. Statistical Properties

In this section we provide parameter estimation error rates for the node conditional estimator in Equation (8). Note that these rates are for the re-parameterized model described in Equation (6)and can be easily translated to guarantees on the original model described in Equations (3), (4).

Notation:

Let ${\mathbb{B}}_2(x,r)=\{y:{\Vert y-x\Vert }_2\le r\}$ be the ℓ₂ ball with center x and radius r. Let ${\{B_t^{\ast }(\cdot )\}}_{t\in V}$ be the true functions of the re-parametrized model, which we would like to estimate from the data. Denote the basis expansion coefficients of B_t(·) with respect to orthonormal basis ${\{{\varphi }_k(\cdot )\}}_{k=0}^{\infty }$ by α_t, which is an infinite dimensional vector and let ${\alpha }_{\mathrm{t}}^{\ast }$ be the coefficients of $B_t^{\ast }(\cdot )$. And let α_t,m be the coefficients corresponding to the top m basis in the basis expansion of B_t(·). Note that $\int B_t{(X)}^2\mathit{dX}={\Vert {\alpha }_{\mathrm{t}}\Vert }_2^2$. Let α = {α_t}_t∈V and α_m = {α_t,m}_t∈V. Let ${\stackrel{‒}{\mathcal{L}}}_{s,m}({\alpha }_{\mathrm{m}})=\mathbb{E}[{\mathcal{L}}_{s,m}({\alpha }_{\mathrm{m}};{\mathbb{X}}_n)]$ be the population version of the sample loss defined in Equation (8). We will often omit ${\mathbb{X}}_n$ from ${\mathcal{L}}_{s,m}({\alpha }_{\mathrm{m}};{\mathbb{X}}_n)$ when clear from the context. We let (α_t – α_t,m) be the difference between infinite dimensional vector α_t and the vector obtained by appropriately padding α_t,m with zeros. Finally, we define the norm $\mathcal{R}(\cdot )$ as $\mathcal{R}({\alpha }_{\mathrm{m}})={\sum }_{t\in V}{\Vert {\alpha }_{\mathrm{t},\mathrm{m}}\Vert }_2$ and its dual as ${\mathcal{R}}^{\ast }({\alpha }_{\mathrm{m}})={\text{sup}}_{t\in V}{\Vert {\alpha }_{\mathrm{t},\mathrm{m}}\Vert }_2$. The norms on infinite dimensional vector α are similarly defined.

We now state our key assumption on the loss function ${\mathcal{L}}_{s,m}(\cdot )$. This assumption imposes strong curvature condition on ${\mathcal{L}}_{s,m}$ along certain directions in a ball around ${\alpha }_{\mathrm{m}}^{\ast }$.

Assumption 1. There exists r_m > 0 and constants c, κ > 0 such that for any ${\Delta }_{\mathrm{m}}\in {\mathbb{B}}_2(0,r_m)$ the gradient of the sample loss ${\mathcal{L}}_{s,m}$ satisfies: $\langle \nabla {\mathcal{L}}_{s,m}({\alpha }_{\mathrm{m}}^{\ast }+{\Delta }_{\mathrm{m}})-\nabla {\mathcal{L}}_{s,m}({\alpha }_{\mathrm{m}}^{\ast }),{\Delta }_{\mathrm{m}}\rangle \ge \kappa {\Vert {\Delta }_{\mathrm{m}}\Vert }_2^2-c\sqrt{\frac{m\log (p)}{n}}\mathcal{R}({\Delta }_{\mathrm{m}})$.

Similar assumptions are increasingly common in analysis of non-convex estimators, see [19] and references therein. We are now ready to state our results which give the parameter estimation error rates, the proofs of which can be found in Appendix. We first provide a deterministic bound on the error ${\Vert {\alpha }_{\mathrm{m}}-{\alpha }_{\mathrm{m}}^{\ast }\Vert }_2$ in terms of the random quantity ${\mathcal{R}}^{\ast }(\nabla {\mathcal{L}}_{s,m}({\alpha }_{\mathrm{m}}^{\ast }))$. We derive probabilistic results in the subsequent corollaries.

Theorem 2. Let ${\mathcal{N}}_s$ be the true neighborhood of node s, with $\mid {\mathcal{N}}_s\mid =d$. Suppose ${\mathcal{L}}_{s,m}$ satisfies Assumption 1. If the regularization parameter λ_n is chosen such that ${\lambda }_n\ge 2{\mathcal{R}}^{\ast }(\nabla {\mathcal{L}}_{s,m}({\alpha }_{\mathrm{m}}^{\ast }))+2c\sqrt{\frac{m\log (p)}{n}}$, then any stationary point ${\widehat{\alpha }}_{\mathrm{m}}$ of (8) in ${\mathbb{B}}_2({\alpha }_{\mathrm{m}}^{\ast },r_m)$ satisfies:

$${\Vert {\widehat{\alpha }}_{\mathrm{m}}-{\alpha }_{\mathrm{m}}^{\ast }\Vert }_2\le \frac{6\sqrt{2}}{\kappa }\sqrt{d}{\lambda }_n.$$

We now provide a set of sufficient conditions under which the random quantity ${\mathcal{R}}^{\ast }(\nabla {\mathcal{L}}_{s,m}({\alpha }_{\mathrm{m}}^{\ast }))$ can be bounded.

Assumption 2. There exists a constant L > 0 such that the gradient of the population loss ${\stackrel{‒}{\mathcal{L}}}_{s,m}$ at ${\alpha }_{\mathrm{m}}^{\ast }$ satisfies: ${\mathcal{R}}^{\ast }(\nabla {\stackrel{‒}{\mathcal{L}}}_{s,m}({\alpha }_{\mathrm{m}}^{\ast }))\le L{\mathcal{R}}^{\ast }({\alpha }^{\ast }-{\alpha }_{\mathrm{m}}^{\ast })$.

Corollary 3. Suppose the conditions in Theorem 2 are satisfied. Moreover, let $\gamma ={\text{sup}}_{i\in \mathbb{N},X\in \mathcal{X}}\mid {\varphi }_i(X)\mid$ and ${\tau }_m={\text{sup}}_{t\in V,X\in \mathcal{X}}\mid {\sum }_{i=1}^m{\alpha }_{t,i}^{\ast }{\varphi }_i(X)\mid$. Suppose ${\mathcal{L}}_{s,m}$ satisfies Assumption 2. If the regularization parameter λ_n is chosen such that ${\lambda }_n\ge 2L{\mathcal{R}}^{\ast }({\alpha }^{\ast }-{\alpha }_{\mathrm{m}}^{\ast })+c\gamma {\tau }_m\sqrt{\frac{{\mathit{md}}^2\log (p)}{n}}$, then then with probability at least 1 – 2m/p² any stationary point ${\widehat{\alpha }}_{\mathrm{m}}$ of (8) in ${\mathbb{B}}_2({\alpha }_{\mathrm{m}}^{\ast },r_m)$ satisfies:

$${\Vert {\widehat{\alpha }}_{\mathrm{m}}-{\alpha }_{\mathrm{m}}^{\ast }\Vert }_2\le \frac{6\sqrt{2}}{\kappa }\sqrt{d}{\lambda }_n.$$

Theorem 2 and Corollary 3 bound the error of the estimated coefficients in the truncated expansion. The approximation error of the truncated expansion itself depends on the function space assumption, as well as the basis chosen, but can be simply combined with the statement of the above corollary to derive the overall error. As an instance, we present a corollary below for the specific case of Sobolev space of order two, and the trigonometric basis.

Corollary 4. Suppose the conditions in Corollary 3 are satisfied. Moreover, suppose the true functions $B_t^{\ast }(\cdot )$ lie in a Sobolev space of order two. Let ${\left\{{\varphi }_k\right\}}_{k=0}^{\infty }$ be the trigonometric basis of $L^2(\mathcal{X})$. If the optimization problem (8) is solved with λ_n = c₁(d² log(p)/n)^2/5 and m = c₂(n/d² log(p))^1/5, then with probability at least 1 – 2m/p² any stationary point ${\widehat{\alpha }}_{\mathrm{m}}$ of (8) in ${\mathbb{B}}_2({\alpha }_{\mathrm{m}}^{\ast },r_m)$ satisfies:

$${\Vert {\widehat{\alpha }}_{\mathrm{m}}-{\alpha }^{\ast }\Vert }_2\le c_3{\left(\frac{d^{13∕4}\log (p)}{n}\right)}^{2∕5},$$

where c₁, c₂, c₃ depend on L, κ, γ, τ_m.

Discussion on Assumption 1:

We now provide a set of sufficient conditions which ensure the restricted strong convexity (RSC) condition. Suppose the population risk ${\stackrel{‒}{\mathcal{L}}}_{s,m}(\cdot )$ is strongly convex in a ball of radius r_m around ${\alpha }_{\mathrm{m}}^{\ast }$

$$\langle \nabla {\stackrel{‒}{\mathcal{L}}}_{s,m}({\alpha }_{\mathrm{m}}^{\ast }+{\Delta }_{\mathrm{m}})-\nabla {\stackrel{‒}{\mathcal{L}}}_{s,m}({\alpha }_{\mathrm{m}}^{\ast }),{\Delta }_{\mathrm{m}}\rangle \ge \kappa {\Vert {\Delta }_{\mathrm{m}}\Vert }_2^2\forall {\Delta }_m\in {\mathbb{B}}_2(0,r_m).$$ (9)

Moreover, suppose the empirical gradients converge uniformly to the population gradients

$$\underset{{\alpha }_{\mathrm{m}}\in {\mathbb{B}}_2({\alpha }_{\mathrm{m}}^{\ast },r_m)}{\text{sup}}{\mathcal{R}}^{\ast }(\nabla {\mathcal{L}}_{s,m}({\alpha }_{\mathrm{m}})-\nabla {\stackrel{‒}{\mathcal{L}}}_{s,m}({\alpha }_{\mathrm{m}}))\le c\sqrt{\frac{m\log p}{n}}.$$ (10)

For example, this condition holds with high probability when the gradient of ${\mathcal{L}}_{s,m}({\alpha }_{\mathrm{m}})$ w.r.t α_t,m, for any t ∈ [p] is a sub-Gaussian process. Equations (9),(10) are easier to check and en sure that ${\mathcal{L}}_{s,m}({\alpha }_{\mathrm{m}})$ satisfies the RSC property in Assumption 1.

6. Connections to Exponential Family MRF Copulas

The Expxorcist class of models could be viewed as being closely related to an exponential family MRF [28] copula density. Consider the parametric exponential family MRF joint density in (3): ${\mathbb{P}}_{\text{MRF};\theta }(X)\propto \text{exp}\left\{{\sum }_{s\in V}{\theta }_s{B}_s(X_s)+{\sum }_{(s,t)\in E(G)}{\theta }_{\mathit{st}}{B}_s(X_s)B_t(X_t)+{\sum }_{s\in V}{C}_s(X_s)\right\}$, where the distribution is indexed by the finite-dimensional parameters {θ_s}_s∈V, {θ_st}_(s,t)∈E, and where in contrast to the previous sections, we assume we are given the sufficient statistics functions {B_s(·)}_s∈V as well as the nodewise base measures {C_s(·)}_s∈V. Now consider the following non-parametric problem. Given a random vector X, suppose we are interested in estimating monotonic node-wise functions {f_s(X_s)}_s∈V such that (f₁(X₁), …, f_p(X_p)) follows ${\mathbb{P}}_{\text{MRF};\theta }$ for some θ. Letting f(X) = (f₁(X₁), …, f_p(X_p)), we have that $\mathbb{P}(\mathbf{f}(X))={\mathbb{P}}_{\text{MRF};\theta }(\mathbf{f}(X))$, so that the density of X can be written as $\mathbb{P}(X)\propto \mathbb{P}(\mathbf{f}(X)){\prod }_{s\in V}{f}_s^{\prime }(X_s)$. This is now a semi-parametric estimation problem, where the unknowns are the functions {f_s(X_s)}_s∈V as well as the finite-dimensional parameters θ. To simplify this density, suppose we assume that the given node-wise sufficient statistics are linear, so that B_s(z) = z, for all s ∈ V, so that density reduces to

$$\mathbb{P}(X)\propto \text{exp}\left\{\sum _{s\in V}{\theta }_s{f}_s(X_s)+\sum _{(s,t)\in E(G)}{\theta }_{\mathit{st}}{f}_s(X_s)f_t(X_t)+\sum _{s\in V}(C_s(f_s(X_s))+\log f_s^{\prime }(X_s))\right\}.$$ (11)

In contrast, the Expxorcist nonparametric exponential family graphical model takes the form

$$\mathbb{P}(X)\propto \text{exp}\left\{\sum _{s\in V}{\theta }_s{f}_s(X_s)+\sum _{(s,t)\in E(G)}{\theta }_{\mathit{st}}{f}_s(X_s)f_t(X_t)+\sum _{s\in V}{C}_s(X_s)\right\}.$$ (12)

It can be seen that the two densities have very similar forms, except that the density in (11) has a more complex base measure that depends on the unknown functions {f_s}_s∈V and importantly the functions {f_s}_s∈V in (11) are monotonic.

The class of densities in (11) can be cast as an exponential family MRF copula density. Suppose we denote the CDF of the parametric exponential family MRF joint density by F_MRF;θ (X), with nodewise marginal CDFs F_MRF;θ,s(X_s). Then the marginal CDF of the density (11) can be written as $F_s(x_s)=\mathbb{P}[X_s\le x_s]=\mathbb{P}[f_s(X_s)\le f_s(x_s)]=F_{\text{MRF};\theta ,s}(f_s(x_s))$, so that

$$f_s(x_s)=F_{\text{MRF};\theta ,s}^{-1}(F_s(x_s)).$$ (13)

It then follows that: $F(X)=F_{\text{MRF};\theta }\left(F_{\text{MRF};\theta ,1}^{-1}(F_1(X_1)),\dots ,F_{\text{MRF};\theta ,p}^{-1}(F_p(X_p))\right)$, where F (X) is the CDF of density (11). By letting $F_{\text{COP};\theta }(U)=F_{\text{MRF};\theta }\left(F_{\text{MRF};\theta ,1}^{-1}(U_1),\dots ,F_{\text{MRF};\theta ,p}^{-1}(U_p)\right)$ be the exponential family MRF copula density function, we see that the CDF of X is precisely: F (X) = F_COP;θ (F₁(X₁), …, F_p(X_p)), which is specified by the marginal CDFs {F_s(X_s)}_s∈V and the copula density F_COP;θ corresponding to the exponential family MRF density. In other words, the non-parametric extension in (11) of the exponential family MRF densities is precisely an exponential family MRF copula density. This development thus generalizes the non-parametric extension of Gaussian MRF densities via the Gaussian copula nonparanormal densities [17]. The caveats with the copula density however are two-fold: the node-wise functions are restricted to be monotonic, but also the estimation of these as in (13) requires the estimation of inverses of marginal CDFs of an exponential family MRF, which is intractable in general. Thus, minor differences in the expressions of the Expxorcist density (12) and an exponential family MRF copula density (11) nonetheless have seemingly large consequences for tractable estimation of these densities from data.

7. Experiments

We present experimental results on both synthetic and real datasets. We compare our estimator, Expxorcist, with the Nonparanormal model of [17] and Gaussian Graphical Model (GGM). We use glasso [7] to estimate GGM and the two step estimator of [17] to estimate Nonparanormal model.

7.1. Synthetic Experiments

Data:

We generated synthetic data from the Expxorcist model with chain and grid graph structures. For both the graph structures, we set θ_s = 1, ∀s ∈ V, θ_st = 1, ∀(s, t) ∈ E and fix the domain $\mathcal{X}$ to [−1, 1]. We experimented with two choices for sufficient statistics B_s(X): sin(4πX) and [exp (−20(X – 0.5)² + exp (−20(X + 0.5)² – 1] and picked the log base measure C_s(X) to be 0. The grid graph we considered has a 10 × (p/10) structure. We used Gibbs sampling to sample data from these models. We also generated data from Gaussian distribution with chain and grid graph structures. To generate this data we set the off diagonal non-zero entries of inverse covariance matrix to 0.49 for chain graph and 0.25 for grid graph and diagonal entries to 1.

Evaluation Metric:

We compared the performance of Expxorcist against baselines, on graph structure recovery, using ROC curves. The ROC curve plots the true positive rate (TPR) against false positive rate (FPR) over different choices of regularization parameter, where TPR is the fraction of correctly detected edges and FPR is the fraction of mis-identified non edges.

Experiment Settings:

For this experiment we set p = 50 and n ∈ {100, 200, 500} and varied the regularization parameter λ from 10−2 to 1. To fit the data to the non parametric model (3), we used cosine basis and truncated the basis expansion to top 30 terms. In practice, one could choose the number of basis (m) based on domain knowledge (e.g. “smooth” functions), or in the absence of which, one could use hold-out validation/cross validation. Given $\widehat{N}(s)$, the estimated neighborhood for node s, we estimated the overall graph structure as: ${\cup }_{s\in V}{\cup }_{t\in \widehat{N}(s)}\{(s,t)\}$. To reduce the variance in the ROC plots, we averaged results over 10 repetitions.

Results:

Figure 1 shows the ROC plots obtained from this experiment. Due to the lack of space, we present more experimental results in Appendix. It can be seen that Expxorcist has much better performance on non-Gaussian data. On these datasets, even at n = 500 the baselines chose edges at random. This suggests that in the presence of multiple modes and fat tails, Expxorcist is a better model. Expxorcist has slightly poor performance than baselines on Gaussian data. However, this is expected because it learns a broader family of distributions than Nonparanormal.

Figure 1:

ROC plots from synthetic experiments. Top and bottom rows show plots for chain and grid graphs respectively. Left column shows plots for data generated from our non-parametric model with B_s(X) = sin(X), n = 500 and center column shows plots for the other choice of sufficient statistic with n = 500. Right column shows plots for Gaussian data with n = 200.

7.2. Futures Intraday Data

We now present our analysis on the Futures price returns. This dataset was downloaded from http://www.kibot.com/. We focus on the Top-26 most liquid instruments being traded at the Chicago Mercantile Exchange (CME). The instruments span different sectors like Energy, Agriculture, Currencies, Equity Indices, Metals and Interest Rates. We focus on the hours of maximum liquidity (9am Eastern to 3pm Eastern) and look at the 1 minute price returns. The return distribution is a mixture of 1 minute returns with the overnight return. Since overnight returns tend to be bigger than the 1 minute return within the day, the return distribution is multimodal and fat-tailed. We treat each instrument as a random variable and the 1 minute returns as independent samples drawn from these random variables. We use the data collected in February 2010 as training data and data from March 2010 as held out data for tuning parameter selection. After removing samples with missing entries we are left with 894 training and 650 held out data samples. We fit Expxorcist and baselines on this data with the same parameter settings described above. For each of these models, we select the best tuning parameter through log likelihood on held out data. However, this criteria resulted in complete graphs for Nonparanormal and GGM (325 edges) and a relatively sparser graph for Expxorcist (168 edges). So for a better comparison of these models, we selected tuning parameters for each of the models such that the resulting graphs have almost the same number of edges. Figure 2 shows the learned graphs for one such choice of tuning parameters, which resulted in ~ 52 edges in the graphs. Nonparanormal and GGM resulted in very similar graphs, so we only present Nonparanormal here. It can be seen that Expxorcist is able to identify the clusters better than Nonparanormal. More detailed graphs and comparison with GGM can be found in Appendix.

Figure 2:

Graph Structures learned for the Futures Intraday Data. The Expxorcist graph shown here was obtained by selecting λ = 0.1. Nodes are colored based on their categories. Edge thickness is proportional to the magnitude of the interaction.

8. Conclusion

In this work we considered the problem of non-parametric density estimation and introduced Expxorcist, a new family of non-parametric graphical models. Our approach relies on a simple function space assumption that the conditional distribution of each variable conditioned on the other variables has a non-parametric exponential family form. We proposed an estimator for Expxorcist that is computationally efficient and comes with statistical guarantees. Our empirical results suggest that, in the presence of multiple modes and fat tails in the data, our non-parametric model is a better choice than the Nonparanormal model of [17].

Supplementary material for Nonparametric Graphical Models Via Conditional Exponential Densities

A. Node Conditional Maximum Likelihood Estimation

In this section we derive the relation between optimization problems (5) and (7) defined in Section 4 of the main paper. We start with the optimization problem (5), which is defined in terms of the original parameters of non-parametric graphical model (3):

$$\begin{array}{c}\hfill \underset{{\mathrm{\Theta }}_s,B}{\text{min}}{\mathcal{L}}_s({\mathrm{\Theta }}_s,B;{\mathbb{X}}_n)+{\lambda }_n{\Vert {\mathrm{\Theta }}_s\Vert }_1\hfill \\ \hfill \text{s.t.}{\theta }_s\ge 0,{\int }_{{\mathcal{X}}_t}{B}_t(X)\mathit{dX}=0,{\int }_{{\mathcal{X}}_t}{B}_t{(X)}^2\mathit{dX}=1\forall t\in V,\hfill \end{array}$$

where ${\mathcal{L}}_s({\mathrm{\Theta }}_s,B;{\mathbb{X}}_n)$ is the node conditional negative log likelihood at node s:

$${\mathcal{L}}_s({\mathrm{\Theta }}_s,B;{\mathbb{X}}_n)=\frac{1}{n}\sum _{i=1}^n\left\{-B_s(X_s^{(i)})\left({\theta }_s+\sum _{t\in V\backslash s}{\theta }_{\mathit{st}}{B}_t(X_t^{(i)})\right)+A(X_{-s}^{(i)};{\theta }_s,B)\right\}.$$

Let ${\tilde{B}}_t(X_t)={\theta }_{\mathit{st}}{B}_t(X_t),\forall t\in V\backslash \left\{s\right\}$. Using this parametrization ${\mathcal{L}}_s(\cdot )$ can be written as:

$${\mathcal{L}}_s({\theta }_s,B_s,{\tilde{B}}_{-s};{\mathbb{X}}_n)=\frac{1}{n}\sum _{i=1}^n\left\{-B_s(X_s^{(i)})\left({\theta }_s+\sum _{t\in V\backslash s}{\tilde{B}}_t(X_t^{(i)})\right)+A(X_{-s}^{(i)};{\mathrm{\Theta }}_s,B_s,{\tilde{B}}_{-s})\right\}.$$

Note that, given ${\tilde{B}}_t$, one can recover θ_st (and thus B_t) by computing the $L^2({\mathcal{X}}_t)$ norm of ${\tilde{B}}_t$. Using this re-parametrization the original optimization in Equation (5) can be written as the following equivalent problem:

$$\begin{array}{c}\hfill \underset{{\theta }_s,B_s,{\tilde{B}}_{-s}}{\text{min}}{\mathcal{L}}_s({\theta }_s,B_s,{\tilde{B}}_{-s};{\mathbb{X}}_n)+{\lambda }_n\sum _{t\in V\backslash s}\sqrt{{\int }_{{\mathcal{X}}_t}{\tilde{B}}_t{(X)}^2\mathit{dX}}+{\lambda }_n\mid {\theta }_s\mid \hfill \\ \hfill \text{s.t.}{\theta }_s\ge 0,{\int }_{{\mathcal{X}}_s}{B}_s{(X)}^2\mathit{dX}=1,{\int }_{{\mathcal{X}}_t}{\tilde{B}}_t(X)\mathit{dX}=0\forall t\in V.\hfill \end{array}$$

The above problem still has the equality constraint on the norm of functions B_s(·). As pointed out in the main paper, this makes the above optimization problem a difficult one to solve. To make the optimization more amenable for numerical optimization techniques, we solve a closely related optimization problem. At each node s ∈ V, we consider the following re-parametrization of B: B_s(X_s) ← θ_sB_s(X_s), B_t(X_t) (θ_st/θ_s)B_t(X_t)∀t ∈ V \ {s}. With a slight abuse of notation we redefine ${\mathcal{L}}_s$ using this re-parametrization as:

$${\mathcal{L}}_s(B;{\mathbb{X}}_n)=\frac{1}{n}\sum _{i=1}^n\left\{-B_s(X_s^{(i)})\left(1+\sum _{t\in V\backslash s}{B}_t(X_t^{(i)})\right)+A(X_{-s}^{(i)};B)\right\},$$

where A(X_−s; B) is the log partition function. We solve the following optimization problem, which is closely related to the original optimization in Equation (5):

$$\begin{array}{c}\hfill \underset{B}{\text{min}}{\mathcal{L}}_s(B;{\mathbb{X}}_n)+{\lambda }_n\sum _{t\in V}\sqrt{{\int }_{{\mathcal{X}}_t}{B}_t{(X)}^2\mathit{dX}}\hfill \\ \hfill \text{s.t.}{\int }_{{\mathcal{X}}_t}{B}_t(X)\mathit{dX}=0\forall t\in V.\hfill \end{array}$$

Note that ${\theta }_s^{\ast }$ need to be bounded away from 0, for this re-parametrized optimization problem to give consistent estimates of the true parameters.

B. Statistical Properties

B.1. Proof of Theorem 2

Proof. Define $\mathbb{C}=\{{\alpha }_{\mathrm{m}}^{\ast }+\Delta :\mathcal{R}({\Delta }_{{\mathcal{N}}_s^c})\le 3\mathcal{R}({\Delta }_{{\mathcal{N}}_s})\}$, where ${\Delta }_{{\mathcal{N}}_s}$ is the sub-vector of Δ restricted to the coordinates specified by variables $\{{\alpha }_{\mathrm{t},\mathrm{m}}:t\in {\mathcal{N}}_s\cup \{s\left\}\right\}$. Let ${\mathcal{F}}_{s,m}$ denote the optimization objective in Equation (8):

$${\mathcal{F}}_{s,m}({\alpha }_{\mathrm{m}};{\mathbb{X}}_n)={\mathcal{L}}_{s,m}({\alpha }_{\mathrm{m}};{\mathbb{X}}_n)+{\lambda }_n\mathcal{R}({\alpha }_{\mathrm{m}}).$$

We prove the theorem in two parts. In the first part we show that ${\mathcal{F}}_{s,m}$ doesn’t have any stationary points in ${\mathbb{B}}_2({\alpha }_{\mathrm{m}}^{\ast },r_m)\cap {\mathbb{C}}^c$. In the second part we show that ${\mathcal{F}}_{s,m}$ doesn’t have any stationary points in ${\mathbb{B}}_2({\alpha }_{\mathrm{m}}^{\ast },r_m)\cap \mathbb{C}\backslash {\mathbb{B}}_2({\alpha }_{\mathrm{m}}^{\ast },r_s)$, where $r_s=(6\sqrt{2}∕\kappa )\sqrt{d}{\lambda }_n$. The proof of the Theorem then follows by combining the results from these two parts.

(a). No stationary points in ${\mathbb{B}}_2({\alpha }_{\mathrm{m}}^{\ast },r_m)\cap {\mathbb{C}}^c$:

Let ${\alpha }_{\mathrm{m}}\in {\mathbb{B}}_2({\alpha }_{\mathrm{m}}^{\ast },r_m)\cap {\mathbb{C}}^c$ and let $\Delta ={\alpha }_{\mathrm{m}}-{\alpha }_{\mathrm{m}}^{\ast }$. Let ∂f(x) denote the set of sub-gradients of a function f(.) at x. For any $z({\alpha }_{\mathrm{m}})\in \partial {\mathcal{F}}_{s,m}({\alpha }_{\mathrm{m}})$, where $z({\alpha }_{\mathrm{m}})=\nabla {\mathcal{L}}_{s,m}({\alpha }_{\mathrm{m}})+{\lambda }_{n}v({\alpha }_{\mathrm{m}})$, $v({\alpha }_{\mathrm{m}})\in \partial \mathcal{R}({\alpha }_{\mathrm{m}})$, we have:

$$\begin{array}{cc}\hfill \langle z({\alpha }_{\mathrm{m}}),{\alpha }_{\mathrm{m}}-{\alpha }_{\mathrm{m}}^{\ast }\rangle =& \langle z({\alpha }_{\mathrm{m}}),\Delta \rangle \hfill \\ \hfill =& \langle \nabla {\mathcal{L}}_{s,m}({\alpha }_{\mathrm{m}}),\Delta \rangle +{\lambda }_n\langle v({\alpha }_{\mathrm{m}}),\Delta \rangle \hfill \\ \hfill =& \langle \nabla {\mathcal{L}}_{s,m}({\alpha }_{\mathrm{m}})-\nabla {\mathcal{L}}_{s,m}({\alpha }_{\mathrm{m}}^{\ast }),\Delta \rangle +\langle \nabla {\mathcal{L}}_{s,m}({\alpha }_{\mathrm{m}}^{\ast }),\Delta \rangle +{\lambda }_n\langle v({\alpha }_{\mathrm{m}}),\Delta \rangle ,\hfill \end{array}$$ (1)

We now bound each of the terms in the RHS of above equation. From Assumption 1 on RSC property of the sample loss, we have

$$\langle \nabla {\mathcal{L}}_{s,m}({\alpha }_{\mathrm{m}})-\nabla {\mathcal{L}}_{s,m}({\alpha }_{\mathrm{m}}^{\ast }),\Delta \rangle \ge \kappa {\Vert \Delta \Vert }_2^2-c\sqrt{\frac{m\log (p)}{n}}\mathcal{R}(\Delta ).$$

From the definition of ${\mathcal{N}}_s$ we have ${({\alpha }_{\mathrm{m}}^{\ast })}_{{\mathcal{N}}_s^c}=0$. So we have

$$\langle v({\alpha }_{\mathrm{m}}),\Delta \rangle =\langle v{({\alpha }_{\mathrm{m}})}_{{\mathcal{N}}_s^c},{({\alpha }_{\mathrm{m}})}_{{\mathcal{N}}_s^c}\rangle +\langle v{({\alpha }_{\mathrm{m}})}_{{\mathcal{N}}_s},{\Delta }_{{\mathcal{N}}_s}\rangle \ge \mathcal{R}({\Delta }_{{\mathcal{N}}_s^c})-\mathcal{R}({\Delta }_{{\mathcal{N}}_s}),$$

where the last inequality follows from the properties of sub-gradient of the norm $\mathcal{R}(.)$. Finally, from the definition of dual norm ${\mathcal{R}}^{\ast }$, we have:

$$\langle \nabla {\mathcal{L}}_{s,m}({\alpha }_{\mathrm{m}}^{\ast }),\Delta \rangle \ge -{\mathcal{R}}^{\ast }(\nabla {\mathcal{L}}_{s,m}({\alpha }_{\mathrm{m}}^{\ast }))\mathcal{R}(\Delta ).$$

Substituting these results in Equation (1) we get:

$$\begin{array}{cc}\hfill \langle z({\alpha }_{\mathrm{m}}),{\alpha }_{\mathrm{m}}-{\alpha }_{\mathrm{m}}^{\ast }\rangle \ge & \kappa {\Vert \Delta \Vert }_2^2-{\mathcal{R}}^{\ast }(\nabla {\mathcal{L}}_{s,m}({\alpha }_{\mathrm{m}}^{\ast }))\mathcal{R}(\Delta )\hfill \\ \hfill & -\left(c\sqrt{\frac{m\log (p)}{n}}\right)\mathcal{R}(\Delta )+{\lambda }_n(\mathcal{R}({\Delta }_{{\mathcal{N}}_s^c})-\mathcal{R}({\Delta }_{{\mathcal{N}}_s})).\hfill \end{array}$$ (2)

Note that since ${\alpha }_{\mathrm{m}}\in {\mathbb{C}}^c$ we have $\mathcal{R}({\Delta }_{{\mathcal{N}}_s^c})\ge 3\mathcal{R}({\Delta }_{{\mathcal{N}}_s})$. Moreover, from our choice of λ_n we know that ${\lambda }_n\ge 2{\mathcal{R}}^{\ast }(\nabla {\mathcal{L}}_{s,m}({\alpha }_{\mathrm{m}}^{\ast }))+2c\sqrt{\frac{m\log (p)}{n}}$. Substituting this in the above equation we get:

$$\langle z({\alpha }_{\mathrm{m}}),{\alpha }_{\mathrm{m}}-{\alpha }_{\mathrm{m}}^{\ast }\rangle \ge \kappa {\Vert \Delta \Vert }_2^2+(\mathcal{R}({\Delta }_{{\mathcal{N}}_s^c})-3\mathcal{R}({\Delta }_{{\mathcal{N}}_s}))\left({\mathcal{R}}^{\ast }(\nabla {\mathcal{L}}_{s,m}({\alpha }_{\mathrm{m}}^{\ast }))+c\sqrt{\frac{m\log (p)}{n}}\right).$$ (3)

The above inequality shows that $\langle z({\alpha }_{\mathrm{m}}),{\alpha }_{\mathrm{m}}-{\alpha }_{\mathrm{m}}^{\ast }\rangle >0$, $\forall {\alpha }_{\mathrm{m}}\in {\mathbb{B}}_2({\alpha }_{\mathrm{m}}^{\ast },r_m)\cap {\mathbb{C}}^c$.

Now suppose ${\alpha }_{\mathrm{m}}\in {\mathbb{B}}_2({\alpha }_{\mathrm{m}}^{\ast },r_m)\cap {\mathbb{C}}^c$ is a stationary point. Then from first-order necessary conditions we know that $\langle z({\alpha }_{\mathrm{m}}),{\alpha }_{\mathrm{m}}^{\ast }-{\alpha }_{\mathrm{m}}\rangle \ge 0$. However, this contradicts the result we obtained in Equation (3). This shows that there are no stationary points in ${\mathbb{B}}_2({\alpha }_{\mathrm{m}}^{\ast },r_m)\cap {\mathbb{C}}^c$.

(b). No stationary points in ${\mathbb{B}}_2({\alpha }_{\mathrm{m}}^{\ast },r_m)\cap \mathbb{C}\backslash {\mathbb{B}}_2({\alpha }_{\mathrm{m}}^{\ast },r_s)$:

The proof of this part follows along the lines of the previous part. Let ${\alpha }_{\mathrm{m}}\in {\mathbb{B}}_2({\alpha }_{\mathrm{m}}^{\ast },r_m)\cap \mathbb{C}\backslash {\mathbb{B}}_2({\alpha }_{\mathrm{m}}^{\ast },r_s)$. From Equations (1), (2) we know that:

$$\begin{array}{cc}\hfill \langle z({\alpha }_{\mathrm{m}}),{\alpha }_{\mathrm{m}}-{\alpha }_{\mathrm{m}}^{\ast }\rangle \ge & \kappa {\Vert \Delta \Vert }_2^2-{\mathcal{R}}^{\ast }(\nabla {\mathcal{L}}_{s,m}({\alpha }_{\mathrm{m}}^{\ast }))\mathcal{R}(\Delta )\hfill \\ \hfill & -\left(c\sqrt{\frac{m\log (p)}{n}}\right)\mathcal{R}(\Delta )+{\lambda }_n\langle v({\alpha }_{\mathrm{m}}),\Delta \rangle .\hfill \end{array}$$ (4)

Since ${\alpha }_{\mathrm{m}}\in \mathbb{C}$ we have $\mathcal{R}(\Delta )\le 4\sqrt{d+1}{\Vert \Delta \Vert }_2\le 4\sqrt{2d}{\Vert \Delta \Vert }_2$. Substituting this in the above equation we get:

$$\begin{array}{cc}\hfill \langle z({\alpha }_{\mathrm{m}}),{\alpha }_{\mathrm{m}}-{\alpha }_{\mathrm{m}}^{\ast }\rangle \ge & \kappa {\Vert \Delta \Vert }_2^2-{\mathcal{R}}^{\ast }(\nabla {\mathcal{L}}_{s,m}({\alpha }_{\mathrm{m}}^{\ast }))\mathcal{R}(\Delta )-\left(c\sqrt{\frac{m\log (p)}{n}}\right)\mathcal{R}(\Delta )-{\lambda }_n\mathcal{R}(\Delta )\hfill \\ \hfill \ge & \kappa {\Vert \Delta \Vert }_2^2-\mathcal{R}(\Delta )\left({\mathcal{R}}^{\ast }(\nabla {\mathcal{L}}_{s,m}({\alpha }_{\mathrm{m}}^{\ast }))+c\sqrt{\frac{m\log (p)}{n}}+{\lambda }_n\right)\hfill \\ \hfill \ge & \kappa {\Vert \Delta \Vert }_2^2-4\sqrt{2d}{\Vert \Delta \Vert }_2\left({\mathcal{R}}^{\ast }({\mathcal{L}}_{s,m}({\alpha }_{\mathrm{m}}^{\ast }))+c\sqrt{\frac{m\log (p)}{n}}+{\lambda }_n\right)\hfill \\ \hfill \ge & \left(\kappa {\Vert \Delta \Vert }_2-4\sqrt{2d}\left({\mathcal{R}}^{\ast }(\nabla {\mathcal{L}}_{s,m}({\alpha }_{\mathrm{m}}^{\ast }))+c\sqrt{\frac{m\log (p)}{n}}+{\lambda }_n\right)\right){\Vert \Delta \Vert }_2\hfill \\ \hfill \ge & \left(\kappa {\Vert \Delta \Vert }_2-4\sqrt{2d}\left(\frac{3}{2}{\lambda }_n\right)\right){\Vert \Delta \Vert }_2.\hfill \end{array}$$ (5)

The above inequality shows that $\langle z({\alpha }_{\mathrm{m}}),{\alpha }_{\mathrm{m}}-{\alpha }_{\mathrm{m}}^{\ast }\rangle >0$, $\forall {\alpha }_{\mathrm{m}}\in {\mathbb{B}}_2({\alpha }_{\mathrm{m}}^{\ast },r_m)\cap \mathbb{C}\backslash {\mathbb{B}}_2({\alpha }_{\mathrm{m}}^{\ast },r_s)$. This shows that there are no stationary points in ${\mathbb{B}}_2({\alpha }_{\mathrm{m}}^{\ast },r_m)\cap \mathbb{C}\backslash {\mathbb{B}}_2({\alpha }_{\mathrm{m}}^{\ast },r_s)$.

Following results from parts (a) and (b) we conclude that any stationary point in ${\mathbb{B}}_2({\alpha }_{\mathrm{m}}^{\ast },r_m)$ satisfies ${\Vert {\alpha }_{\mathrm{m}}-{\alpha }_{\mathrm{m}}^{\ast }\Vert }_2\le \frac{6\sqrt{2}}{\kappa }\sqrt{d}{\lambda }_n$.□

B.2. Proof of Corollary 3

Before we proceed to the proof of Corollary 3, we introduce some notation we use in its proof. We say Z is a σ–sub-Gaussian random variable, if it satisfies the following tail property:

$$\mathbb{P}(\mid Z-\mathbb{E}[Z]\mid \ge ϵ)\le 2\text{exp}\{-\frac{{ϵ}^2}{2{\sigma }^2}\}.$$

We use the notation Z ∈ SG(σ²) to say that a random variable Z is σ–sub-Gaussian. We use the following standard result from concentration theory: if Z₁ … Z_n are n i.i.d SG(σ²) random variables then $\frac{1}{n}{\sum }_{i=1}^n{Z}_i\in \mathit{SG}\left(\frac{{\sigma }^2}{n}\right)$.

The following Lemma provides an upper bound for ${\mathcal{R}}^{\ast }(\nabla {\mathcal{L}}_{s,m}({\alpha }_{\mathrm{m}}^{\ast }))$ that holds with high probability.

Lemma 1. Let ${\mathcal{N}}_s$ be the true neighborhood of node s, with $\mid {\mathcal{N}}_s\mid =d$. Moreover, let

$$\gamma =\underset{i\in \mathbb{N},X\in \mathcal{X}}{\text{sup}}\mid {\varphi }_i(X)\mid$$

and

$${\tau }_m=\underset{t\in V,X\in \mathcal{X}}{\text{sup}}\mid \sum _{i=1}^m{\alpha }_{t,i}^{\ast }{\varphi }_i(X)\mid .$$

Suppose ${\mathcal{L}}_{s,m}$ satisfies Assumption 2. Then with probability at least 1 – 2m/p² we have:

$${\mathcal{R}}^{\ast }(\nabla {\mathcal{L}}_{s,m}({\alpha }_{\mathrm{m}}^{\ast }))\le L{\mathcal{R}}^{\ast }({\alpha }^{\ast }-{\alpha }_{\mathrm{m}}^{\ast })+c_1\gamma {\tau }_m\sqrt{\frac{{\mathit{md}}^2\log (p)}{n}},$$

where c₁ > 0 is a constant.

Proof. From triangle inequality, we have:

$${\mathcal{R}}^{\ast }(\nabla {\mathcal{L}}_{s,m}({\alpha }_{\mathrm{m}}^{\ast }))\le {\mathcal{R}}^{\ast }(\nabla {\mathcal{L}}_{s,m}({\alpha }_{\mathrm{m}}^{\ast })-\nabla {\stackrel{‒}{\mathcal{L}}}_{s,m}({\alpha }_{\mathrm{m}}^{\ast }))+{\mathcal{R}}^{\ast }(\nabla {\stackrel{‒}{\mathcal{L}}}_{s,m}({\alpha }_{\mathrm{m}}^{\ast })).$$

We now upper bound each of the terms in the RHS of the above equation.

(a). Upper bound for ${\mathcal{R}}^{\ast }(\nabla {\mathcal{L}}_{s,m}({\alpha }_{\mathrm{m}}^{\ast })-\nabla {\stackrel{‒}{\mathcal{L}}}_{s,m}({\alpha }_{\mathrm{m}}^{\ast }))$:

The gradient of ${\mathcal{L}}_{s,m}({\alpha }_{\mathrm{m}})$ at ${\alpha }_{\mathrm{m}}^{\ast }$ is given by:

$${(\nabla {\mathcal{L}}_{s,m}({\alpha }_{\mathrm{m}}^{\ast }))}_{s,k}=\frac{1}{n}\sum _{i=1}^n-\left({\varphi }_k(X_s^{(i)})E_{-s}(X_{-s}^{(i)};{\alpha }_{\mathrm{m}}^{\ast })\right)+{\mathbb{E}}_{{\alpha }_{\mathrm{m}}^{\ast }}\left[\phantom{\mid }{\varphi }_k(X_s)E_{-s}(X_{-s}^{(i)};{\alpha }_{\mathrm{m}}^{\ast })\mid X_{-s}^{(i)}\right]$$

and for t ∈ V \ {s}:

$${(\nabla {\mathcal{L}}_{s,m}({\alpha }_{\mathrm{m}}^{\ast }))}_{t,k}=\frac{1}{n}\sum _{i=1}^n-\left({\varphi }_k(X_t^{(i)})B_s(X_s^{(i)};{\alpha }_{\mathrm{m}}^{\ast })\right)+{\mathbb{E}}_{{\alpha }_{\mathrm{m}}^{\ast }}\left[\phantom{\mid }{\varphi }_k(X_t^{(i)})B_s(X_s;{\alpha }_{\mathrm{m}}^{\ast })\mid X_{-s}^{(i)}\right],$$

where $E_{-s}(X_{-s};{\alpha }_{\mathrm{m}})=1+{\sum }_{t\in V\backslash \left\{s\right\}}{\sum }_{j=1}^m{\alpha }_{t,j}{\varphi }_j(X_t)$ and $B_s(X_s;{\alpha }_{\mathrm{m}})={\sum }_{j=1}^m{\alpha }_{s,j}{\varphi }_j(X_s)$ and ${\mathbb{E}}_{\alpha }[.]$ denotes expectation with respect to the density parametrized by α and $(\nabla {\mathcal{L}}_{s,m}{({\alpha }_{\mathrm{m}}^{\ast }))}_{t,k}$ is the gradient of ${\mathcal{L}}_{s,m}({\alpha }_{\mathrm{m}})$ evaluated at ${\alpha }_{\mathrm{m}}^{\ast }$ with respect to variable α_t,k.

We now show that $(\nabla {\mathcal{L}}_{s,m}{({\alpha }_{\mathrm{m}}^{\ast }))}_{t,k}$ concentrates well around its expectation. Note that

$$\mathbb{E}\left[(\nabla {\mathcal{L}}_{s,m}{({\alpha }_{\mathrm{m}}^{\ast }))}_{t,k}\right]={(\nabla {\stackrel{‒}{\mathcal{L}}}_{s,m}({\alpha }_{\mathrm{m}}^{\ast }))}_{t,k}.$$

Lets first define random variable ${\left\{Y_{s,m}\right\}}_{k=1}^m$ and ${\left\{Y_{t,k}\right\}}_{k=1}^m,\forall t\in V\backslash \left\{s\right\}$ as:

$$Y_{s,k}(X^{(i)})=-\left({\varphi }_k(X_s^{(i)})E_{-s}(X_{-s}^{(i)};{\alpha }_{\mathrm{m}}^{\ast })\right)+{\mathbb{E}}_{{\alpha }_{\mathrm{m}}^{\ast }}\left[\phantom{\mid }{\varphi }_k(X_s)E_{-s}(X_{-s}^{(i)};{\alpha }_{\mathrm{m}}^{\ast })\mid X_{-s}^{(i)}\right],$$

and

$$Y_{t,k}(X^{(i)})=-\left({\varphi }_k(X_t^{(i)})B_s(X_s^{(i)};{\alpha }_{\mathrm{m}}^{\ast })\right)+{\mathbb{E}}_{{\alpha }_{\mathrm{m}}^{\ast }}\left[\phantom{\mid }{\varphi }_k(X_t^{(i)})B_s(X_s;{\alpha }_{\mathrm{m}}^{\ast })\mid X_{-s}^{(i)}\right].$$

To ease the notation, we denote Y_s,k(X⁽ⁱ⁾) by $Y_{s,k}^{(i)}$. We now rewrite $\nabla {\mathcal{L}}_{s,m}({\alpha }_{\mathrm{m}}^{\ast })$ in terms of random variables ${\left\{Y_{s,k}\right\}}_{k=1}^m$ and ${\left\{Y_{t,k}\right\}}_{k=1}^m$ as follows:

$${(\nabla {\mathcal{L}}_{s,m}({\alpha }_{\mathrm{m}}^{\ast }))}_{s,k}=\frac{1}{n}\sum _{i=1}^n{Y}_{s,k}^{(i)},{(\nabla {\mathcal{L}}_{s,m}({\alpha }_{\mathrm{m}}^{\ast }))}_{t,k}=\frac{1}{n}\sum _{i=1}^n{Y}_{t,k}^{(i)}.$$

For any k ∈ [1, m], i ∈ [1, n] we have:

$$\begin{array}{cc}\hfill \mid Y_{s,k}^{(i)}\mid \le & \mid \left({\varphi }_k(X_s^{(i)})E_{-s}(X_{-s}^{(i)};{\alpha }_{\mathrm{m}}^{\ast })\right)\mid +\mid \phantom{[}{\mathbb{E}}_{{\alpha }_{\mathrm{m}}^{\ast }}[{\varphi }_k(X_s)E_{-s}(X_{-s}^{(i)};{\alpha }_{\mathrm{m}}^{\ast })\mid X_{-s}^{(i)}]\mid \hfill \\ \hfill \le & \gamma \mid \left(E_{-s}(X_{-s}^{(i)};{\alpha }_{\mathrm{m}}^{\ast })\right)\mid +\gamma {\mathbb{E}}_{{\alpha }_{\mathrm{m}}^{\ast }}\left[\mid \phantom{\mid }{E}_{-s}(X_{-s}^{(i)};{\alpha }_{\mathrm{m}}^{\ast })\mid \mid X_{-s}^{(i)}\right],\hfill \end{array}$$

where the last inequality follows from Jensen’s inequality and the fact that $\gamma =\underset{j\in \mathbb{N},X\in \mathcal{X}}{\text{sup}}\mid {\varphi }_j(X)\mid$.

We now bound $\mid \left(E_{-s}(X_{-s}^{(i)};{\alpha }_{\mathrm{m}}^{\ast })\right)\mid$:

$$\begin{array}{cc}\hfill \mid \left(E_{-s}(X_{-s}^{(i)};{\alpha }_{\mathrm{m}}^{\ast })\right)\mid =& \mid 1+\sum _{t\in V\backslash \left\{s\right\}}\sum _{j=1}^m{\alpha }_{t,j}^{\ast }{\varphi }_j(X_t^{(i)})\mid \hfill \\ \hfill \le & 1+\sum _{t\in V\backslash \left\{s\right\}}\mid \sum _{j=1}^m{\alpha }_{t,j}^{\ast }{\varphi }_j(X_j^{(i)})\mid \hfill \\ \hfill \le & 1+d{\tau }_m\le d(1+{\tau }_m).\hfill \end{array}$$

Substituting this in the above equation we get:

$$\mid Y_{s,k}^{(i)}\mid \le 2\gamma d(1+{\tau }_m).$$

Using similar arguments we can show that $\mid Y_{t,k}^{(i)}\mid \le 2\gamma {\tau }_m$, ∀t ∈ V \ {s}. This shows that $Y_{s,k}^{(i)}\in \text{SG}(4{\gamma }^2{d}^2{(1+{\tau }_m)}^2)$ and $Y_{t,k}^{(i)}\in \text{SG}(4{\gamma }^2{\tau }_m^2)$, ∀t ∈ V \ {s}. Since ${(\nabla {\mathcal{L}}_{s,m}({\alpha }_{\mathrm{m}}^{\ast }))}_{t,k}$ is a sum of n i.i.d random variables ${\left\{Y_{t,k}^{(i)}\right\}}_{i=1}^n$ we have:

$${(\nabla {\mathcal{L}}_{s,m}({\alpha }_{\mathrm{m}}^{\ast }))}_{s,k}\in \text{SG}\left(\frac{4{\gamma }^2{d}^2{(1+{\tau }_m)}^2}{n}\right),{(\nabla {\mathcal{L}}_{s,m}({\alpha }_{\mathrm{m}}^{\ast }))}_{t,k}\in \text{SG}\left(\frac{4{\gamma }^2{\tau }_m^2}{n}\right).$$

From the concentration properties of a sub-Gaussian random variable we have:

$$\mathbb{P}\left(\mid {(\nabla {\mathcal{L}}_{s,m}({\alpha }_{\mathrm{m}}^{\ast }))}_{s,k}-{(\nabla {\stackrel{‒}{\mathcal{L}}}_{s,m}({\alpha }_{\mathrm{m}}^{\ast }))}_{s,k}\mid \le ϵ\right)\ge 1-2\text{exp}\left\{-\frac{n{ϵ}^2}{8{\gamma }^2{(1+{\tau }_m)}^2{d}^2}\right\}.$$

and

$$\mathbb{P}\left(\mid {(\nabla {\mathcal{L}}_{s,m}({\alpha }_{\mathrm{m}}^{\ast }))}_{t,k}-{(\nabla {\stackrel{‒}{\mathcal{L}}}_{s,m}({\alpha }_{\mathrm{m}}^{\ast }))}_{t,k}\mid \le ϵ\right)\ge 1-2\text{exp}\left\{-\frac{n{ϵ}^2}{8{\gamma }^2{\tau }_m^2}\right\},\forall t\in V\backslash \{s\}.$$

Now let ${(\nabla {\mathcal{L}}_{s,m}({\alpha }_{\mathrm{m}}^{\ast }))}_t={\left\{{(\nabla {\mathcal{L}}_{s,m}({\alpha }_{\mathrm{m}}^{\ast }))}_{t,k}\right\}}_{k=1}^m$. By union bound we get:

$$\begin{array}{cc}\hfill & \mathbb{P}\left({\Vert {(\nabla {\mathcal{L}}_{s,m}({\alpha }_{\mathrm{m}}^{\ast }))}_s-{(\nabla {\stackrel{‒}{\mathcal{L}}}_{s,m}({\alpha }_{\mathrm{m}}^{\ast }))}_s\Vert }_2\ge ϵ\right)\hfill \\ \hfill & \le \sum _{k=1}^m\mathbb{P}\left(\mid {(\nabla {\mathcal{L}}_{s,m}({\alpha }_{\mathrm{m}}^{\ast }))}_{s,k}-{(\nabla {\stackrel{‒}{\mathcal{L}}}_{s,m}({\alpha }_{\mathrm{m}}^{\ast }))}_{s,k}\mid \ge \frac{ϵ}{\sqrt{m}}\right)\hfill \\ \hfill & \le 2m\text{exp}\left\{-\frac{n{ϵ}^2}{8{\gamma }^2{(1+{\tau }_m)}^2{\mathit{md}}^2}\right\}\hfill \end{array}$$

and

$$\mathbb{P}\left({\Vert {(\nabla {\mathcal{L}}_{s,m}({\alpha }_{\mathrm{m}}^{\ast }))}_t-{(\nabla {\stackrel{‒}{\mathcal{L}}}_{s,m}({\alpha }_{\mathrm{m}}^{\ast }))}_t\Vert }_2\ge ϵ\right)\le 2m\text{exp}\left\{-\frac{n{ϵ}^2}{8{\gamma }^2{\tau }_m^{2}m}\right\}.$$

By using union bound again we get:

$$\begin{array}{cc}\hfill \mathbb{P}({\mathcal{R}}^{\ast }(\nabla {\mathcal{L}}_{s,m}({\alpha }_{\mathrm{m}}^{\ast })-\nabla {\stackrel{‒}{\mathcal{L}}}_{s,m}({\alpha }_{\mathrm{m}}^{\ast }))\ge ϵ)=& \mathbb{P}\left(\underset{t\in V}{\text{sup}}\Vert {(\nabla {\mathcal{L}}_{s,m}({\alpha }_{\mathrm{m}}^{\ast }))}_t-{(\nabla {\stackrel{‒}{\mathcal{L}}}_{s,m}({\alpha }_{\mathrm{m}}^{\ast }))}_t\Vert \ge ϵ\right)\hfill \\ \hfill \le & 2m\text{exp}\left\{-\frac{n{ϵ}^2}{8{\gamma }^2{(1+{\tau }_m)}^2{\mathit{md}}^2}\right\}\hfill \\ \hfill & +2m(p-1)\text{exp}\left\{-\frac{n{ϵ}^2}{8{\gamma }^2{\tau }_m^{2}m}\right\}\hfill \\ \hfill \le & 2\mathit{mp}\text{exp}\left\{-\frac{n{ϵ}^2}{8{\gamma }^2{(1+{\tau }_m)}^2{\mathit{md}}^2}\right\}.\hfill \end{array}$$

Choosing $ϵ=\sqrt{24{\gamma }^2{(1+{\tau }_m)}^2{\mathit{md}}^2\log (p)∕n}$, we get the following: with probability at least

$${\mathcal{R}}^{\ast }(\nabla {\mathcal{L}}_{s,m}({\alpha }_{\mathrm{m}}^{\ast })-\nabla {\stackrel{‒}{\mathcal{L}}}_{s,m}({\alpha }_{\mathrm{m}}^{\ast }))\le \sqrt{24{\gamma }^2{(1+{\tau }_m)}^2{\mathit{md}}^2\log (p)∕n}.$$

(b). Upper bound for ${\mathcal{R}}^{\ast }(\nabla {\stackrel{‒}{\mathcal{L}}}_{s,m}({\alpha }_{\mathrm{m}}^{\ast }))$:

From Assumption 2 we have the following upper bound for ${\mathcal{R}}^{\ast }(\nabla {\stackrel{‒}{\mathcal{L}}}_{s,m}({\alpha }_{\mathrm{m}}^{\ast }))$:

$${\mathcal{R}}^{\ast }(\nabla {\stackrel{‒}{\mathcal{L}}}_{s,m}({\alpha }_{\mathrm{m}}^{\ast }))\le L{\mathcal{R}}^{\ast }({\alpha }_{\mathrm{m}}^{\ast }-{\alpha }^{\ast }).$$

Combining the results from parts (a) and (b) we get the following: with probability at least 1–2m/p²

$${\mathcal{R}}^{\ast }(\nabla {\stackrel{‒}{\mathcal{L}}}_{s,m}({\alpha }_{\mathrm{m}}^{\ast }))\le L{\mathcal{R}}^{\ast }({\alpha }_{\mathrm{m}}^{\ast }-{\alpha }^{\ast })+c_1\gamma {\tau }_m\sqrt{{\mathit{md}}^2\log (p)∕n},$$

where c₁ > 0 is a constant.□

Following results from Theorem 2, Lemma 1, we conclude that if the regularization parameter λ_n is chosen such that

$${\lambda }_n\ge 2L{\mathcal{R}}^{\ast }({\alpha }_{\mathrm{m}}^{\ast }-{\alpha }^{\ast })+c_1\gamma {\tau }_m\sqrt{{\mathit{md}}^2\log (p)∕n},$$

then with probability at least 1 – 2m/^p2, any stationary point α_m in ${\mathbb{B}}_2({\alpha }_{\mathrm{m}}^{\ast },r_m)$ satisfies

$${\Vert {\alpha }_{\mathrm{m}}-{\alpha }_{\mathrm{m}}^{\ast }\Vert }_2\le \frac{6\sqrt{2}}{\kappa }\sqrt{d}{\lambda }_n.$$

B.3. Proof of Corollary 4

Let α_m be any stationary point in ${\mathbb{B}}_2({\alpha }_{\mathrm{m}}^{\ast },r_m)$. And suppose λ_n is chosen such that:

$${\lambda }_n=2L{\mathcal{R}}^{\ast }({\alpha }_{\mathrm{m}}^{\ast }-{\alpha }^{\ast })+c_1\gamma {\tau }_m\sqrt{{\mathit{md}}^2\log (p)∕n}.$$

In this section we derive bounds for the overall estimation error ∥α_m – α*∥₂. From triangle inequality, we have:

$${\Vert {\alpha }_{\mathrm{m}}-{\alpha }^{\ast }\Vert }_2\le {\Vert {\alpha }_{\mathrm{m}}-{\alpha }_{\mathrm{m}}^{\ast }\Vert }_2+{\Vert {\alpha }_{\mathrm{m}}^{\ast }-{\alpha }^{\ast }\Vert }_2.$$

From Theorem 2, we have a bound for ${\Vert {\alpha }_{\mathrm{m}}-{\alpha }_{\mathrm{m}}^{\ast }\Vert }_2$. So, here we focus on bounding ${\Vert {\alpha }_{\mathrm{m}}^{\ast }-{\alpha }^{\ast }\Vert }_2$.

Since the true parameters $B_t^{\ast }(.)$ lie in a Sobolev space of order two, we know that there exists a constant c₂ > 0 such that [1]:

$$\underset{t\in V}{\text{sup}}{\Vert {\alpha }_{\mathrm{t},\mathrm{m}}^{\ast }-{\alpha }_{\mathrm{t}}^{\ast }\Vert }_2\le \frac{c_2}{m^2}.$$

Combining this result with the result from Theorem 2, we get, with probability at least 1 – 2m/p²:

$$\begin{array}{cc}\hfill {\Vert {\alpha }_{\mathrm{m}}-{\alpha }^{\ast }\Vert }_2\le & {\Vert {\alpha }_{\mathrm{m}}-{\alpha }_{\mathrm{m}}^{\ast }\Vert }_2+{\Vert {\alpha }_{\mathrm{m}}^{\ast }-{\alpha }^{\ast }\Vert }_2\hfill \\ \hfill \le & \frac{6\sqrt{2}}{\kappa }\sqrt{d}{\lambda }_n+\frac{c_2\sqrt{d}}{m^2}\hfill \\ \hfill =& \frac{6\sqrt{2}}{\kappa }\sqrt{d}\left(2L{\mathcal{R}}^{\ast }({\alpha }_{\mathrm{m}}^{\ast }-{\alpha }^{\ast })+c_1\gamma {\tau }_m\sqrt{{\mathit{md}}^2\log (p)∕n}\right)+\frac{c_2\sqrt{d}}{m^2}\hfill \\ \hfill \le & \frac{6\sqrt{2}}{\kappa }\sqrt{d}\left(\frac{2c_{2}L}{m^2}+c_1\gamma {\tau }_m\sqrt{{\mathit{md}}^2\log (p)∕n}\right)+\frac{c_2\sqrt{d}}{m^2}\hfill \\ \hfill \le & c_3\sqrt{d}\left[\frac{L}{\kappa }\frac{1}{m^2}+\gamma {\tau }_m\sqrt{{\mathit{md}}^2\log (p)∕n}\right],\hfill \end{array}$$

where c₃ > 0 is a constant. Choosing $m={\left(\frac{L}{\kappa \gamma {\tau }_m}\right)}^{2∕5}{\left(\frac{n}{d^2\log (p)}\right)}^{1∕5}$ gives us the following error bound:

$${\Vert {\alpha }_{\mathrm{m}}-{\alpha }^{\ast }\Vert }_2\le c_4{\left(\frac{L{\gamma }^4{\tau }_m^4}{\kappa }\right)}^{1∕5}{\left(\frac{d^{13∕4}\log (p)}{n}\right)}^{2∕5},$$

and the corresponding λ_n for this choice of m is given by:

$${\lambda }_n=c_5{\left(\frac{L{\gamma }^4{\tau }_m^4}{\kappa }\right)}^{1∕5}{\left(\frac{d^2\log (p)}{n}\right)}^{2∕5},$$

where c₄, c₅ > 0 are constants.

Figure 1:

ROC plots for data generated from non-parametric graphical model with B_s(X) =sin(4πX). Top row shows results for chain graph with C_s(X) = 0. Bottom row shows results for grid graph with C_s(X) = 0.

C. Experiments

C.1. Synthetic Data

In this section we present additional results from our synthetic experiments. Specifically, we present ROC plots for n ∈ {100, 200, 500}. We use the same parameter settings as described in Section 7 of the main paper, to generate synthetic data. Figure 1 shows ROC plots for data generated from non-parametric graphical model with B_s(X) = sin(4πX). Figure 2 shows ROC plots for B_s(X) = [exp (−20(X – 0.5)²) + exp (−20(X + 0.5)²) – 1] and Figure 3 shows ROC plots for Gaussian data.

C.2. Futures Intraday Data

In this section we present the graph learned by GGM for Futures Intraday data and also present more detailed graphs learned by all the three estimators. As pointed out in Section 7, selecting tuning parameter based on held out log likelihood resulted in very dense graphs for Nonparanormal and GGM. So we use a different technique to compare all the models. We fix a tuning parameter for Expxorcist and select tuning parameters for the baselines that resulted in graphs with same number of edges. Figure 4 shows the graph structures learned for one such choice of tuning parameters. Figures 5, 7, 6 present more detailed graphs for the corresponding graphs in Figure 4.

Figure 2.

ROC plots for data generated from non-parametric graphical model with B_s(X) = [exp (−20(X – 0.5)²) + exp (−20(X + 0.5)²) – 1]. Top row shows results for chain graph with C_s(X) = 0. Bottom row shows results for grid graph with C_s(X) = 0.

Figure 3:

ROC plots for data generated from Gaussian distributions. Top row shows plots for chain graph and bottom row shows plots for grid graph.

Figure 4:

Graph Structures learned for the Futures Intraday Data. The Expxorcist graph shown here was obtained by selecting λ = 0.1. Nodes are colored based on their categories. Edge thickness is proportional to the magnitude of the interaction.

Figure 5:

Nonparanormal.

Figure 6:

Expxorcist.

Figure 7:

GGM.