Graph Laplacian Learning with Exponential Family Noise

Abstract

Graph signal processing (GSP) is a prominent framework for analyzing signals on non-Euclidean domains. The graph Fourier transform (GFT) uses the combinatorial graph Laplacian matrix to reveal the spectral decomposition of signals in the graph frequency domain. However, a common challenge in applying GSP methods is that in many scenarios the underlying graph of a system is unknown. A solution in such cases is to construct the unobserved graph from available data, which is commonly referred to as graph or network inference. Although different graph inference methods exist, they are restricted to learning from either smooth graph signals or simple additive Gaussian noise. Other types of noisy data, such as discrete counts or binary digits, are rather common in real-world applications, yet are underexplored in graph inference. In this paper, we propose a versatile graph inference framework for learning from graph signals corrupted by exponential family noise. Our framework generalizes previous methods from continuous smooth graph signals to various data types. We propose an alternating algorithm that jointly estimates the graph Laplacian and the unobserved smooth representation from the noisy signals. We also extend our approach to include an offset variable which models different levels of variation of the nodes. Since real-world graph signals are frequently non-independent and temporally correlated, we further adapt our original setting to a timevertex formulation. We demonstrate on synthetic and real-world data that our new algorithms outperform competing Laplacian estimation methods that suffer from noise model mismatch.

I. Introduction

GRAPHS are ubiquitous in our world. From neural systems to molecules, from social networks to traffic flow, objects in assorted fields and their connections can be naturally abstracted as graphs. Studying these abstractions and the data associated with them has been proven vital in understanding these scientific fields [1]-[4].

Graph signal processing (GSP) provides an elegant yet powerful framework for studying these geometric objects and serves as the foundation of many other modern geometric machine learning approaches. GSP analyzes data or signals on irregular domains, generalizing from signals on a regular grid such as time series to signals on networks and graphs [5]. The graph shift operator (GSO), usually a matrix representation of the graph such as the graph adjacency and the graph Laplacian, generalizes the temporal shift operator in traditional signal processing. The eigenvectors of the GSO not only encompass rich geometric information about the graph but also transform spatial signals to the spectral domain. Thus, by making the eigenvectors of the GSO graph Fourier basis, GSP extends Fourier analysis to non-Euclidean domains. Functions of the eigenvalues of GSO are called graph filters, which amplify or dampen different spectral components of graph signals. However, applying the graph Fourier transform (GFT) requires the graph of the system to be known, which is not the case in many real-world problems. When the graph is unknown, one needs to resort to various graph inference methods [6] to apply GSP or other graph machine learning methods.

Graph inference is a prominent problem in GSP. With only graph signals available, learning the graph structure is possible when imposing various geometric priors on the signals [6]-[9]. Arguably the most prevalent is to assume that the observed signals are smooth with respect to the graph to be inferred. Established methods usually search for a desirable graph adjacency matrix or, more commonly, a graph Laplacian, such that the total variation of given signals, a measurement of smoothness, will be minimal [10]-[14]. However, in the real world, one is often required to deal with noisy signals which violate this assumption. The smooth model implies that signals are continuous and unbounded, while in the real world, signals can be bounded, discrete, or even binary. One way to bypass this problem is to disregard the noise and treat the noisy signals as smooth signals, especially when the noise is assumed to be considerably small. Although such compromises may yield decent results, ignoring the noise is not always appropriate. In neuroscience, for example, graph learning methods have been successfully applied to fMRI data to learn functional connectivity between brain regions [10], [15], [16]. However, at the cellular level, electrophysiology recordings consist of binary spiking patterns. The non-continuous and non-negative nature of spiking signals complicates neuron connectivity inference using existing smooth graph methods. Similarly, in microbiome data analysis, non-continuous count data are used to learn microbial interaction networks [17]. Finally, in psychology, one can learn networks between subjects from binary or integer-valued questionnaires that subjects fill out [18], [19]. Thus, new graph learning approaches are necessary when smoothness is not properly defined.

Fortunately, a remedy for the smooth model is feasible, owing to its probabilistic interpretation [11], [20]. For the Laplacian-based smooth model, with the graph Laplacian properly regularized, graph inference coincides with the maximum likelihood estimation (MLE) of an improper Gaussian distribution of smooth signals. More interestingly, Dong et al. [11] considered the case where the noise is additive Gaussian, and formulated graph inference using the maximum a-posterior (MAP) of the unobserved Gaussian smooth signals. In the same vein as generalized linear models (GLM) generalizing linear models, an appropriate noise distribution can substitute the Gaussian distribution in smooth signals to account for a specific response. More precisely, we can model the observations by a conditional response distribution whose expected value is parameterized by their underlying smooth representation. This new noise model can adapt the smooth model to other data types, with the underlying smooth signal kept intact for utilizing GSP.

In this paper, we study the problem of learning graph Laplacian matrices from graph signals of various types. We first generalize a GSP-based framework with Gaussian noise model [11] to a versatile framework compatible with all exponential family noise. We then propose Graph Learning with Exponential family Noise (GLEN), an algorithm that estimates the underlying graph Laplacian from the noisy signals while jointly inferring the unobserved smooth representation. Our generalized method provides a unified solution for graph Laplacian learning where the observed data type can vary. We present two concrete examples of this generalized scenario with Poisson and Binomial noise. Furthermore, we extend GLEN to the time-vertex setting [21] to handle data with temporal correlations, e.g., time series on a network. We demonstrate in synthetic experiments that GLEN is competitive against off-the-shelf Laplacian estimation methods under noise model mismatch. Furthermore, we apply GLEN to different types of real-world data (e.g., binary questionnaires and neural activity) to further demonstrate the efficacy of our methods. Note that our goal is to learn rigorous combinatorial graph Laplacian matrices, which is different from previous work that aims to learn general precision matrices [17], [22].

In summary, our contributions are as follows:

1. We establish a generalized GSP-based framework that models graph signals of different data types using exponential family distributions;

2. We propose an alternating algorithm for joint graph learning and signal denoising under this generalized framework;

3. We extend our framework to temporally smooth signals using the time-vertex formulation.

The rest of the paper is organized as follows. We review related work on graph inference in Section II and introduce the mathematical background in Section III. In Section IV we present a framework to generalize GSP from simple Gaussian noise to exponential family noise and propose GLEN for learning the unknown graph from this framework. In Section V, we extend GLEN to encompass different levels of offsets among the nodes and to include temporal structures on each node. We evaluate our methods on synthetic and real-world datasets in Section VI and conclude in Section VII.

II. Related Work

The problem of graph inference has been studied from three different perspectives: statistical modeling, GSP, and physically motivated models.

In Probabilistic Graphical Models (PGMs), a graph typically refers to the conditional dependencies between random variables. One important class of PGM is the Gaussian Markov random field (GMRF) [23]. In a GMRF, the conditional dependencies are encoded by the inverse covariance matrix, also called the precision matrix. The problem of learning the parsimonious precision matrix is known as covariance selection [24], assuming that the graph to be estimated is sparse. Nowadays, covariance selection is usually solved by the prestigious graphical Lasso algorithm and its different variants [25]-[33]. Although the graph Laplacian also encodes the node dependencies, it is worth mentioning that learning general GMRFs is intrinsically different from the graph Laplacian inference problem in this paper. This is because a general precision matrix can have negative conditional dependencies but a graph in this context cannot contain negative edges. Additionally, since Laplacians are first-order intrinsic, the set of Laplacians is disjoint from the set of general precision, which is nonsingular. Other PGMs, such as different variants of GMRFs, discrete Markov random fields and Bayesian networks, also have found their applications in many fields, such as physics [34], genetics [35], [36], microbial ecology [17], [22], [37] and causal inference [38], [39]. But again, the learned graphs are not Laplacian-constrained.

The graph inference methods in GSP tackle the problem from a different perspective. In GSP, the observations are typically assumed to be the outputs of a graph filtering system with some specific input signals. Inference methods vary on the assumptions of the form of graph filters, i.e. the function of GSO, and the characteristics of initial signals. The smoothness of the signals with respect to the graph is arguably the most popular assumption [11]-[14], [40]-[42]. Heat diffusion filters, also relying on the graph Laplacian, were studied in [43]. When the GSO is the adjacency matrix or the weighted adjacency matrix, (truncated) polynomial filters are widely used [44]-[46]. The normalized graph Laplacian matrix was also explored in [47], and its connection to the normalized adjacency matrix was also discussed. Besides the graph filters, assumptions on the input or transformed signals can also vary. In [48], the input signals are assumed to be sparse instead of standard Gaussian. Recent methods also learn graphs that endow the observations with sparse spectral representations [49], [50].

Interestingly, connections can be made between statistical models and GSP. Although a general precision matrix is intrinsically different from a graph Laplacian, one can purposely construct an improper GMRF to accommodate the Laplacian structures. Such an exception is the class of attractive intrinsic GMRFs, where the precision matrices are improper and restricted to M-matrices so that they share the same properties as combinatorial graph Laplacians. As we show in the next section, fitting this improper GMRF to the data [51] induces the same smoothness measurement term as learning a valid combinatorial graph Laplacian.

We describe our framework using the language of GSP but show that it also admits a PGM-like interpretation. For example, when the noise distribution is selected as Poisson, our framework is closely related to the Poisson-Log-Normal (PLN) model. While graph inference from different data types was explored in the field of PGMs, e.g., the Ising model and the Poisson graphical model [52], it is rarely discussed in the field of GSP where a combinatorial graph Laplacian need to be learned. This paper aims to complete this missing piece of rigorous graph Laplacian learning when noise varies across a broad spectrum of exponential family distributions. Also, because typical PGMs such as PLN do not learn rigorous graph Laplacians, we only focus on GSP methods in this paper. Physically motivated models consider the observations to be outcomes of some physical processes. This is however less related to our setting and we refer the interested readers to the comprehensive surveys on graph inference in [6], [20].

III. Background

A. Preliminaries

We use the following notation throughout the paper. Let lowercase and uppercase bold letters denote vectors and matrices respectively. $\mathbf{S}{\mathbb{R}}^{N\times N}$ and $\mathbf{S}{\mathbb{R}}_{+}^{N\times N}$ denote the set of symmetric real matrices and real semi-definite positive matrices. Let $0$, $1$, and $2$ denote all 0, 1, and 2 vectors, and $\mathbf{I}$ denote the identity matrix. ˝ denotes the Hadamard product. •: denotes the Moore-Penrose pseudo-inverse and det^† denotes the pseudo-determinant. For matrix norms, ${\Vert \cdot \Vert }_F$ denotes the Frobenius norm, ${\Vert \cdot \Vert }_S$ denotes the grand sum of all entries, and ${\Vert \cdot \Vert }_1$ denotes the grand sum of all absolute entry values. We use ${\Vert \cdot \Vert }_2$ to denote vector 2-norm.

Consider a weighted undirected graph $G=\{𝒱,ℰ,\mathbf{W}\}$, where $𝒱$ denotes the set of $N$ vertices, $ℰ$ the set of edges and $\mathbf{W}\in \mathbf{S}{\mathbb{R}}^{N\times N}$ the weighted adjacency matrix. Each entry ${\mathbf{W}}_{ij}={\mathbf{W}}_{ji}\ge 0$, $\forall 1\le i\le j\le N$ corresponds to the weight of edge ${ℰ}_{ij}$. We let ${\mathbf{W}}_{ii}=0,\forall 1\le i\le N$ so there is no self-loop. Let $\mathbf{D}\in \mathbf{S}{\mathbb{R}}^{N\times N}$ denote the diagonal weighted degree matrix, such that the diagonal entry ${\mathbf{D}}_{ii}={\sum }_j{\mathbf{W}}_{ij}$ is the degree of node ${𝒱}_i$. The combinatorial Laplacian matrix $\mathbf{L}$ of graph $G$ is given by $\mathbf{L}=\mathbf{D}-\mathbf{W}$. It is also convenient to define $\mathbf{H}=c{\mathbf{I}}_N-1_N{1}_N^T$, where $c\ge 1$, allowing for different regularization of the form ${\Vert \mathbf{L}\circ \mathbf{H}\Vert }_1$ on $\mathbf{L}$. For instance, when $c=1$ this term applies ${\ell }_1$ regularization on the graph weights, while for $c=2$ it also penalizes the node degrees.

A real graph signal is a function $x:𝒱\to {\mathbb{R}}^N$ that assigns a real value to each graph vertex. The definition of a smooth signal varies across different contexts, but generally, a smooth graph signal $\mathbf{x}$ can be seen as the result of applying a low-pass graph filter $ℱ(\mathbf{L})$ to a non-smooth signal ${\mathbf{x}}_0$

$$\mathbf{x}=ℱ(\mathbf{L}){\mathbf{x}}_0=\sum _i{\mathbf{u}}_{i}f({\lambda }_i){\mathbf{u}}_i^T{\mathbf{x}}_0,$$ (1)

where $\{{\mathbf{u}}_i,{\lambda }_i\}$, $\forall 1\le i\le N$ are the eigenvector-eigenvalue pairs of $\mathbf{L}$. When ${\mathbf{x}}_0~𝒩(0,{\mathbf{I}}_N)$, the smooth signal follows the distribution

$$\mathbf{x}\sim 𝒩(0,ℱ{(\mathbf{L})}^2).$$ (2)

This GSP formulation of applying a low-pass filter to white ${\mathbf{x}}_0$, as well as different choices of the filter have been well-summarized by [12]. A particular choice is $ℱ(\mathbf{L})=\sqrt{{\mathbf{L}}^{†}}$. The smooth signals then follow an improper Gaussian distribution

$$\mathbf{x}\sim 𝒩(0,{\mathbf{L}}^{†}).$$ (3)

B. Graph Inference from Smooth Signals

Consider a dataset $\mathbf{X}=[{\mathbf{x}}_1,{\mathbf{x}}_2,\dots ,{\mathbf{x}}_M]\in {\mathbb{R}}^{N\times M}$, where each row corresponds to a node and each column corresponds to an independent smooth graph signal ${\mathbf{x}}_j\in {\mathbb{R}}^N$. The goal of graph inference is to learn such $G$ that the columns of $\mathbf{X}$ will be smooth on it. Laplacian-based graph learning methods solve the following optimization problem:

$$\underset{\mathbf{L}\in ℒ}{\text{min}}\{\text{Tr}({\mathbf{X}}^T\mathbf{LX})+\alpha h(\mathbf{L})\},$$ (4)

where $ℒ$ is the space of valid graph Laplacian matrices

$$ℒ=\{\mathbf{L}\in {\mathbf{S}\mathbb{R}}_{+}^{N\times N}\mid \mathbf{L}1=0,{\mathbf{L}}_{ij}={\mathbf{L}}_{ji}\le 0,\forall i\ne j\},$$ (5)

$h(\mathbf{L})$ is a regularization term, and $\alpha$ is a trade-off parameter. (4) is consist of two terms. The first term is the graph Laplacian quadratic form, also known as the Dirichlet energy. It measures the overall smoothness of graph signals since $\text{Tr}({\mathbf{X}}^T\mathbf{LX})={\sum }_{k=1}^M{\sum }_{1\le i<j\le N}{\mathbf{W}}_{ij}{({\mathbf{X}}_{ik}-{\mathbf{X}}_{jk})}^2$. Furthermore, consider the system in (1) and choose the low-pass filter $ℱ(\mathbf{L})=\sqrt{{\mathbf{L}}^{†}}$. The smoothness term naturally arises in the density function of (3). These elucidate how (4) imposes the smoothness prior. The second regularization term serves two purposes. Firstly, it prevents trivial solutions such as fully disconnected graphs where $\mathbf{L}$ contains zero everywhere. Secondly, it imposes additional priors on the graph such as sparsity.

In (4), the smoothness term can be formulated in various ways and the specific choice of regularization can also vary. This results in different objectives with different optimization methods. Here we briefly describe some popular graph learning methods and the specific forms of (4) they employ. [10] and [11] use the original smoothness term and choose the regularization term $h(\mathbf{L})$ to be the Frobenius norm of $\mathbf{L}$. A trace equality constraint $\text{Tr}(\mathbf{L})=N$ is added to prevent trivial solutions. They formulate the problem as a quadratic program with linear constraints and solve it through interiorpoint methods. [12] reformulate the smoothness term as $\Vert \mathbf{W}\circ \mathbf{Z}\Vert =\frac{1}{2}\text{Tr}({\mathbf{X}}^T\mathbf{LX})$, where $\mathbf{Z}\in \mathbf{S}{\mathbb{R}}^{N\times N}$ is the squared Euclidean distance matrix across rows of $\mathbf{X}$. They adapt the previous Frobenius norm regularization with a logarithmic barrier to promote connectivity and solve for $\mathbf{W}$ with primal-dual optimization. Solving for $\mathbf{W}$ is equivalent to solving for $\mathbf{L}$. Finally, [13] propose the following objective

$$\underset{\mathbf{L}\in ℒ}{\text{min}}\left\{\text{Tr}(\mathbf{LS})-\log {\text{det}}^{†}(\mathbf{L})+\alpha {\Vert \mathbf{L}\circ \mathbf{H}\Vert }_1\right\},$$ (6)

where $\mathbf{S}$ is a data statistic and the design of $\mathbf{H}$ imposes different regularizations on $\mathbf{L}$. The authors propose to use block coordinate descent on the rows and columns of $\mathbf{L}$ for optimization. To reveal the connection between (4) and (6), let the data statistics be the commonly used sample covariance matrix $\mathbf{S}=\frac{{\mathbf{XX}}^{\mathbf{T}}}{M}$. This makes the first term of (6) a scaled smoothness term since $\text{Tr}(\mathbf{LS})=\frac{1}{M}\text{Tr}({\mathbf{LXX}}^T)=\frac{1}{M}\text{Tr}({\mathbf{X}}^T\mathbf{LX})$. The log-determinant regularization is also an interesting choice because it emerges naturally from the MLE of (3). In fact, (6) is exactly a penalized MLE of the improper GMRF, whose precision matrix is the $\mathbf{L}$.

C. Graph Inference with Gaussian Noise

Consider a more realistic case where the observations $\mathbf{X}$ are noisy. Dong et al. [11] model noisy observations as underlying smooth representations $\mathbf{y}$ corrupted by additive isotropic Gaussian noise $ϵ$. Since smooth representations follow a Gaussian distribution, as shown in Eq. (1), the noisy observations also follow a Gaussian distribution

$$\mathbf{x}\sim 𝒩(0,ℱ{(\mathbf{L})}^2+{\sigma }_{ϵ}^2{\mathbf{I}}_N),$$ (7)

where ${\sigma }_{ϵ}$ is the standard deviation of Gaussian noise $ϵ$. Again let $ℱ(\mathbf{L})=\sqrt{{\mathbf{L}}^{†}}$, the MAP estimation of unobserved smooth representation $\mathbf{y}$ from (7) amounts to

$$\underset{\mathbf{y}}{\text{min}}\left\{{\Vert \mathbf{x}-\mathbf{y}\Vert }_2^2+\beta {\mathbf{y}}^T\mathbf{L}\mathbf{y}\right\},$$ (8)

where hyper-parameter $\beta$ reflects the variance of the noise ${\sigma }_{ϵ}^2$. Based on (8), jointly learning $\mathbf{L}$ and the unobserved smooth signal representations $\mathbf{Y}=[{\mathbf{y}}_1,{\mathbf{y}}_2,\dots ,{\mathbf{y}}_M]\in {\mathbb{R}}^{N\times M}$ was proposed in [11] as

$$\underset{\mathbf{L}\in ℒ,\mathbf{Y}}{\text{min}}\left\{{\Vert \mathbf{X}-\mathbf{Y}\Vert }_F^2+\beta (\text{Tr}({\mathbf{Y}}^T\mathbf{L}\mathbf{Y})+\alpha h(\mathbf{L}))\right\},$$ (9)

where $h(\mathbf{L})$ is the same Frobenius norm and trace equality constraint $\text{Tr}(\mathbf{L})=N$. To solve (9), their method switches between two steps where they fix one variable and solve for the other. The advantage of this optimization scheme is that each sub-problem objective is convex even though (9) is not. When $\mathbf{Y}$ is fixed ($\mathbf{Y}$-step), the problem coincides with the Laplacian learning from smooth signals as in (4). When $\mathbf{L}$ is fixed ($\mathbf{L}$-step), solving for $\mathbf{Y}$

$$\underset{\mathbf{Y}}{\text{min}}\left\{{\Vert \mathbf{X}-\mathbf{Y}\Vert }_F^2+\beta (\text{Tr}({\mathbf{Y}}^T\mathbf{L}\mathbf{Y}))\right\},$$ (10)

enjoys a closed-form solution $\mathbf{Y}={({\mathbf{I}}_N+\beta \mathbf{L})}^{-1}\mathbf{X}$ that amounts to the Tikhonov filtering of $\mathbf{X}$.

IV. Graph Inference from Noisy Signals

In this section, we first present a versatile noise model that accounts for various data types with exponential family distributions. We then propose GLEN, an alternating algorithm that jointly infers the unknown graph and denoises the signals from the exponential family model. Finally, we provide two concrete examples of this generalized scenario, with Poisson and Binomial noise, to conclude this section. Our method is demonstrated in Fig. 1.

Fig. 1.

Graph Laplacian learning with Exponential Family Noise (GLEN). A. Diagram showing how GLEN jointly optimizes the unknown graph $G$ and the unknown latent smooth representation $\mathbf{Y}$ given observations $\mathbf{X}$. GLEN alternates these two steps: (i) a graph learning step that infers $G$ from current $\mathbf{Y}$ estimation and (ii) a denoising step that smooths binary observations $\mathbf{X}$ using the current $G$ to obtain new $\mathbf{Y}$. Each of the $M$ columns of $\mathbf{Y}$ (the red vertical waveform) is a smooth graph signal over $N$ nodes. B. Example of a Bernoulli graph signal $\mathbf{x}$. A solid blue bar indicates a 1 and an empty bar indicates a 0. C. Example of a smooth graph signal $\mathbf{y}$ underlying the noisy Poisson observations. D. Example of temporally correlated smooth graph signals.

A. Exponential Family Noise Model

We first propose a GSP-based framework to model the generative process of noisy signals of different data types, motivated by the Gaussian noise case. Specifically, the underlying smooth representation is generated from the same process as in (1), which is then connected to the mean parameter of the exponential family distribution through a link function $g$

$$\mathbf{E}[\mathbf{x}\mid \mathbf{y}]=g^{-1}(\mathbf{y}),s.t.\mathbf{y}\sim 𝒩(0,ℱ{(\mathbf{L})}^2).$$ (11)

More precisely, we consider exponential family distributions of the following form

$$p(\mathbf{x}\mid \eta )=k(\mathbf{x})\text{exp}\{{\eta }^{T}T(\mathbf{x})-A(\eta )\},$$ (12)

specified by natural parameters $\eta$, sufficient statistics $T(\mathbf{x})$, and the cumulant generating function $A(\eta )$, which corresponds to the normalization factor of the probability distribution. We slightly abuse $T(\cdot )$ and $A(\cdot )$ to denote element-wise operations. Let the density function of $𝒫(\eta )$ be (12). Then the response model of noisy signals is given by

$$\mathbf{x}\sim 𝒫(g^{-1}(\mathbf{y})),\mathbf{y}\sim 𝒩(0,ℱ{(\mathbf{L})}^2).$$ (13)

In reminiscence of generalized linear models (GLMs), we link smooth signals to the mean parameters through the function $g$ so that (11) holds. We list common exponential family distributions in Tab. I.

TABLE I.

Exponential family and link functions

Distribution	$\theta$	$\eta$	$T(\mathbf{x})$	$A(\eta )$	g
Bernouli	$p$	$\text{log}\frac{p}{1-p}$	x	$\text{log}(1+e^{\eta })$	logit
Binomial	$p$	$\text{log}\frac{p}{1-p}$	x	$n\text{log}(1+e^{\eta })$	logit
Negative Binomial	$p$	$\text{log}p$	x	$-r\text{log}(1-e^{\eta })$	logit
Poisson	$\lambda$	$\text{log}\lambda$	x	$e^{\eta }$	log
Gaussian	[$\mu$, ${\sigma }^2$]	[$\mu ∕{\sigma }^2$, $-1∕2{\sigma }^2$]	[$x$, $x^2$]	${\mu }^2∕2$	identity

When $p(\mathbf{x}\mid \mathbf{y})$ takes some specific exponential family distribution, $p(\mathbf{x})$ can find close relatives in the distribution zoo. Let $\mathbf{y}$ follow the same smooth signal distribution in (3). When the response is Poisson and the link function is logarithmic, $p(\mathbf{x})$ can be considered as an improper PLN distribution [53] with the precision matrix constrained to be a combinatorial graph Laplacian

$$\mathbf{x}\sim 𝒫ℒ𝒩(0,{\mathbf{L}}^{†}).$$ (14)

Similarly, one can obtain improper Bernoulli-Logit-Normal distributions or improper Binomial-Logit-Normal distributions [54] from this framework, to name a few. The Laplacian constraints distinguish these improper distributions from their counterparts in the PGM literature and permit a direct connection to GSP that has not been previously addressed.

B. Graph Inference with Exponential Family Noise

We propose GLEN to learn the graph topology from the noisy observations as in (13). Following [11], we consider the MAP estimate of the unobserved smooth representation $\mathbf{y}$

$$\underset{\mathbf{y}}{\text{min}}\left\{-{\mathbf{y}}^{T}T(\mathbf{x})+1_N^{T}A(\mathbf{y})+\frac{1}{2}{\mathbf{y}}^T\mathbf{L}\mathbf{y}\right\}.$$ (15)

The full objective is then derived from the joint estimation of $\mathbf{Y}$ and $\mathbf{L}$

$$\begin{gathered} \underset{\mathbf{Y},\mathbf{L}\in ℒ}{\text{min}}\{-\text{Tr}({\mathbf{Y}}^{T}T(\mathbf{X}))+{\Vert A(\mathbf{Y})\Vert }_S\phantom{\}} \\ \phantom{\{}+\frac{1}{2}\text{Tr}({\mathbf{Y}}^T\mathbf{L}\mathbf{Y})+\frac{\alpha }{2}h(\mathbf{L})\} \\ s.t.{\mathbf{Y}}^T{1}_N=0_M, \end{gathered}$$ (16)

where $h(\mathbf{L})$ regularizes the graph and is up to the user’s choice. Here we slightly abuse the notation of $A$ to denote the element-wise computation. We also add a constraint for $\mathbf{y}$ which we explain below. The first two terms measure the fidelity of the inferred smooth representations $\mathbf{Y}$ with respect to noisy observation $\mathbf{X}$. The last two terms are the same as (4), imposing smoothness and other structural priors on the inferred graph. When the distribution is isotropic Gaussian, (16) coincides with (9). If the regularization is chosen to be the same as (9), one can fully recover the method in [11] for learning the graph from Gaussian observations (Sec. III-C).

GLEN alternates between an $\mathbf{L}$-step and a $\mathbf{Y}$-step to jointly learn these variables from (16), with a specific exponential family distribution as the noise model. At each step, we update the corresponding variables while the other variable is fixed. Similar to [11], we obtain convexity in each sub-problem although the original problem is not jointly convex. Alg. 1 summarizes the full algorithm for general exponential family distributions.

The $\mathbf{L}$-step is akin to the smooth signal learning scenario with $\mathbf{Y}$ fixed. This is because the choice of the exponential family distribution only affects the fidelity terms of (16). We simply replace the noisy $\mathbf{X}$ in (4) with the smooth signal representation $\mathbf{Y}$ to obtain the $\mathbf{L}$-step

$$\underset{\mathbf{L}\in ℒ}{\text{min}}\{\text{Tr}({\mathbf{Y}}^T\mathbf{LY})+\alpha h(\mathbf{L})\}.$$ (17)

Depending on the choice of regularization $h(\mathbf{L})$, previous methods [11]-[13] can be readily plugged in. Here we propose a new regularization inspired by (6)

$$\alpha h(\mathbf{L})=-{\alpha }_1\log {\text{det}}^{†}(\mathbf{L})+{\alpha }_2{\Vert \mathbf{L}\circ \mathbf{H}\Vert }_1.$$ (18)

Here ${\alpha }_1$ and ${\alpha }_2$ are non-negative trade-off parameters. Different from (6), the log-determinant term is also controlled by a trade-off parameter ${\alpha }_1$, which we found useful when dealing with exponential family noise. The resulting optimization problem is still convex and we use gradient descent to solve it.

The $\mathbf{Y}$-step denoises $\mathbf{X}$ using the current $\mathbf{L}$ estimation. When $\mathbf{L}$ is fixed, updating $\mathbf{Y}$ becomes the MAP in (15). However, the solution of (15) might not be meaningful: the fidelity might become dominant and the smoothness term might be simply ignored. Because the combinatorial graph Laplacian $\mathbf{L}$ is first-order intrinsic ${L1}_N=0_N$, the smoothness term cannot penalize large uniform signals when they are favored by the fidelity term. Thus, we solve a constrained optimization of $\mathbf{Y}$

$$\begin{gathered} \underset{\mathbf{Y}}{\text{min}}\left\{-\text{Tr}({\mathbf{Y}}^{T}T(\mathbf{X}))+{\Vert A(\mathbf{Y})\Vert }_S+\frac{1}{2}\text{Tr}({\mathbf{Y}}^T\mathbf{LY})\right\}, \\ s.t.{\mathbf{Y}}^T{1}_N=0_M. \end{gathered}$$ (19)

The constraint prohibits $\mathbf{Y}$ to grow along the direction $1$, therefore striking for a balance between the fidelity and smoothness term. Since (19) does not enjoy a closed-form solution for general exponential family distributions, we resort to iterative methods. We apply the Newton-Raphson method with linear constraints to each smooth signal representation ${\mathbf{y}}_j$. Let the gradient and Hessian of the unconstrained problem be ${\nabla }_{{\mathbf{y}}_j}$ and ${\nabla }_{{\mathbf{y}}_j}^2$

$${\nabla }_{{\mathbf{y}}_j}=-T({\mathbf{x}}_j)+A^{\prime }({\mathbf{y}}_j)+{\mathbf{Ly}}_j,$$ (20)

$${\nabla }_{{\mathbf{y}}_j}^2=\text{diag}(A^{″}({\mathbf{y}}_j))+\mathbf{L}.$$ (21)

Algorithm 1 GLEN

$$\begin{array}{cc}\hfill & \mathrm{Input}:\text{noisy signals}\mathbf{X}=[{\mathbf{x}}_1,{\mathbf{x}}_2,\dots ,{\mathbf{x}}_M],\text{exponential}\hfill \\ \hfill & \text{family distribution}𝒫,\text{trade-off parameters}{\alpha }_1,{\alpha }_2,\text{and}\gamma ,\hfill \\ \hfill & \text{stepsize}\rho \text{and}\lambda \hfill \\ \hfill & \mathrm{Output}:\text{graph Laplacian}\mathbf{L},\text{smooth signal representation}\hfill \\ \hfill & \mathbf{Y}=[{\mathbf{y}}_1,{\mathbf{y}}_2,\dots ,{\mathbf{y}}_M]\hfill \\ \hfill & \text{Initialize}\mathbf{Y}\text{and}\mathbf{L},\text{or}\mathbf{Y},\mathbf{L},\text{and}\mu \text{if using the offset}\hfill \\ \hfill & \mathbf{repeat}\hfill \\ \hfill & \text{Solve}\mathbf{L}\text{in}(17)\text{using gradient descent}\hfill \\ \hfill & \mathbf{for}j=1,2,\dots ,M\mathbf{do}\hfill \\ \hfill & \text{Calculate}{\nabla }_{{\mathbf{y}}_j}\text{and}{\nabla }_{{\mathbf{y}}_j}^2\text{as in}(20)\text{and}(21),(\text{or}(39)\hfill \\ \hfill & \text{and}(40)\text{when using time-vertex signals})\hfill \\ \hfill & \text{Solve}\Delta {\mathbf{y}}_j\text{from}(22)\text{and update}{\mathbf{y}}_j\text{as in}(23)\hfill \\ \hfill & \mathbf{end}\mathbf{for}\hfill \\ \hfill & \text{Update}\mu \text{as in}(33)\text{if using the offset}\hfill \\ \hfill & \text{Calculate the loss using}(16)\hfill \\ \hfill & \mathbf{until}\text{convergence}.\hfill \end{array}$$

We solve the following linear system

$$\left[\begin{array}{cc}\hfill {\nabla }_{{\mathbf{y}}_j}^2\hfill & \hfill 1\hfill \\ \hfill 1^T\hfill & \hfill 0\hfill \end{array}\right]\left[\begin{array}{c}\hfill \Delta {\mathbf{y}}_j\hfill \\ \hfill w\hfill \end{array}\right]=\left[\begin{array}{c}\hfill -{\Delta }_{{\mathbf{y}}_j}\hfill \\ \hfill 0\hfill \end{array}\right],$$ (22)

and update ${\mathbf{y}}_j$

$${\mathbf{y}}_j^{(t)}={\mathbf{y}}_j^{(t-1)}+\rho \Delta {\mathbf{y}}_j.$$ (23)

Here $\rho$ is the stepsize. Because (22) ensures that $1_N^T{\Delta }_{{\mathbf{y}}_j}=0$, $1_N^T{\mathbf{y}}_j^{(t)}=0$ always hold so long as $1_N^T{\mathbf{y}}_j^{(1)}=0$. Note that we use a Cholesky solver to solve the linear equation, avoiding explicit matrix inversion. The Cholesky solver has a similar $𝒪(N^3)$ complexity but is numerically more stable. For the sake of efficiency, we perform only one Newton-Raphson step for each ${\mathbf{y}}_j$ during a $\mathbf{Y}$-step. This reduces computation, especially when $\mathbf{L}$ is poorly learned in the first few alternating steps.

C. Noise Model Examples

Here we present the derivation of the solution for two specific examples with Poisson and Binomial distributions.

1). Derivation of Poisson Observations:

First, we derive the objective functions and update rules for the Poisson distribution. Given Poisson distribution as shown in Tab. I and the generalized objective in Eq. (16), we obtain the graph learning objective with Poisson noise:

$$\begin{gathered} \underset{\mathbf{Y},\mathbf{L}\in ℒ}{\text{min}}\{-\text{Tr}({\mathbf{Y}}^T\mathbf{X})+{\Vert \text{exp}(\mathbf{Y})\Vert }_S\phantom{\}} \\ \phantom{\{}+\frac{1}{2}\text{Tr}({\mathbf{Y}}^T\mathbf{LY})+\frac{\alpha }{2}h(\mathbf{L})\}, \\ s.t.{\mathbf{Y}}^T{1}_N=0_M. \end{gathered}$$ (24)

We abuse $\text{exp}(\cdot )$ to denote element-wise exponential operation. The gradient and Hessian for the unconstrained Newton–Raphson are then given by

$${\nabla }_{{\mathbf{y}}_j}=-{\mathbf{x}}_j+\text{exp}({\mathbf{y}}_j)+{\mathbf{Ly}}_j,$$ (25)

$${\nabla }_{{\mathbf{y}}_j}^2=\text{diag}(\text{exp}({\mathbf{y}}_j))+\mathbf{L}.$$ (26)

The update $\Delta {\mathbf{y}}_j$ is then obtained from Eq. (22).

2). Derivation of Binomial Observation:

The second example uses the Binomial distribution. Given the Binomial distribution as shown in Tab. I with parameter $n=R$, we have

$$\begin{gathered} \underset{\mathbf{Y},\mathbf{L}\in ℒ}{\text{min}}\{-\text{Tr}({\mathbf{Y}}^T\mathbf{X})+R{\Vert \log (1_N{1}_M^T+\text{exp}(\mathbf{Y}))\Vert }_S\phantom{\}} \\ \phantom{\{}+\frac{1}{2}\text{Tr}({\mathbf{Y}}^T\mathbf{LY})+\frac{\alpha }{2}h(\mathbf{L})\}, \\ s.t.{\mathbf{Y}}^T{1}_N=0_M. \end{gathered}$$ (27)

The Newton–Raphson for solving $\mathbf{Y}$ are derived similarly

$${\nabla }_{{\mathbf{y}}_j}=-{\mathbf{x}}_j+\frac{R}{1_N+\text{exp}(-{\mathbf{y}}_j)}+{\mathbf{Ly}}_j,$$ (28)

$${\nabla }_{{\mathbf{y}}_j}^2=\text{diag}\left(\frac{R}{\text{exp}({\mathbf{y}}_j)+2_N+\text{exp}(-{\mathbf{y}}_j)}\right)+\mathbf{L}.$$ (29)

Again, the division is element-wise for the sake of simplicity. If we set $R=1$, then the Bernoulli noise model is obtained.

V. Graph Inference Extensions

We present two extensions of GLEN. First, we add a constant offset $\mu$ to the unobserved smooth signals $\mathbf{Y}$. Next, we present an adaptation to the time-vertex formulation [21] where the data admits a joint graph-temporal structure, i.e. at every time point we observe a graph signal, and at each node of the graph we observe a time-series.

A. Graph Inference with Offsets

In (13), we assume the underlying smooth representation always has zero offset (mean). However, such an assumption is not always favorable for real-world data. For example, when the noise model is Poisson, a non-zero offset is often useful for modeling different levels of exposure to an event. This motivates GLEN to model the offset explicitly when necessary.

For the Gaussian noise model, the offset variable is redundant since graph inference methods can work on the residuals of the observations $\mathbf{X}-\frac{1}{M}{X1}_M{1}_M^T$. For general exponential family noise, however, the offset contributes to the responses non-linearly, and accounting for its effects in GLEN is nontrivial. To resolve this issue, we add an offset term $\mu$ to (16) which is jointly inferred with $\mathbf{Y}$ and $\mathbf{L}$

$$\begin{gathered} \underset{\mathbf{Y},\mu ,\mathbf{L}\in ℒ}{\text{min}}\{-\text{Tr}(({\mathbf{Y}}^T+1_M{\mu }^T)T(\mathbf{X}))+{\Vert A(\mathbf{Y}+\mu 1_M^T)\Vert }_S\phantom{\}} \\ \phantom{\{}+\frac{1}{2}\text{Tr}({\mathbf{Y}}^T\mathbf{LY})+\frac{\alpha }{2}h(\mathbf{L})\}, \\ s.t.{\mathbf{Y}}^T{1}_N=0_M. \end{gathered}$$ (30)

and the $\mathbf{Y}$-step needs to be slightly modified

$${\nabla }_{{\mathbf{y}}_j}=-T({\mathbf{x}}_j)+A^{\prime }({\mathbf{y}}_j+\mu )+{\mathbf{Ly}}_j,$$ (31)

$${\nabla }_{{\mathbf{y}}_j}^2=\text{diag}(A^{″}({\mathbf{y}}_j+\mu ))+\mathbf{L},$$ (32)

The $\mathbf{Y}$ constraints here also prevent infinite solutions. Now we need to infer the offset variable $\mu$. Similar to the $\mathbf{Y}$-step, we use unconstrained Newton-Raphson for the $\mu$-step

$${\mu }^{(t)}={\mu }^{(t-1)}+{\lambda [{\nabla }_{\mu }^2]}^{-1}{\nabla }_{\mu },$$ (33)

where $\lambda$ is the stepsize and

$${\nabla }_{\mu }=\sum _j-T({\mathbf{x}}_j)+A^{\prime }({\mathbf{y}}_j+\mu ),$$ (34)

$${\nabla }_{\mu }^2=\sum _j\text{diag}(A^{″}({\mathbf{y}}_j+\mu )).$$ (35)

The extended GLEN cycles through these three steps to learn the Laplacian.

B. Graph Inference from Time Series

In many real-world applications, graph signals, i.e., the columns of the input matrix $\mathbf{X}$, are not always independent. A common case is that the graph signals have a natural temporal order. In such a scenario, each row of the matrix, corresponding to a graph node, is a canonical 1-D time series, and $\mathbf{X}$ can be seen as a multivariate time-varying signal lying on the graph. For example, in neural data analysis, the input is often a count matrix whose rows correspond to neurons and columns correspond to time bins. While a graph on the rows models the functional connectivity of neurons, the columns should also vary smoothly across time. Such an assumption of temporal correlation merges classical signal processing with GSP and proves useful in many scenarios (e.g., neuroscience, sensor networks).

The Time-Vertex signal processing framework [21], [55] is specifically designed for this setting. For a time-varying graph signal $\mathbf{X}\in {\mathbb{R}}^{N\times M}$, the time-vertex GSP is defined on the Cartesian product of a graph $G$ that underlies the $N$ rows and a temporal graph $T$ that underlies the $M$ columns. The Laplacian quadratic form of a matrix signal $\mathbf{X}$ on the product graph $J$ is the summation of the quadratic forms along each dimension

$$\text{vec}{(\mathbf{X})}^T{\mathbf{L}}_J\text{vec}(\mathbf{X})=\text{Tr}({\mathbf{X}}^T{\mathbf{L}}_G\mathbf{X})+\text{Tr}({\mathbf{XL}}_T{\mathbf{X}}^T)$$ (36)

where $\text{vec}(\mathbf{X})$ is the column-wise vectorization of $\mathbf{X}$, and ${\mathbf{L}}_J$, ${\mathbf{L}}_G$, ${\mathbf{L}}_T$ denote the Laplacians of $J$, $G$, $T$, respectively. A smooth time-vertex signal is smooth on both factor graphs as measured in (36). The joint time-vertex Fourier transform (JFT) is simply applying the graph Fourier transform (GFT) on the $G$ dimension and the discrete Fourier transform (DFT) on the $T$ dimension.

Here we assume the underlying representation of the input matrix is smooth on the time-vertex graph $J$. As the temporal graph $T$ is known a priori (typically modeled as a path graph), we still need to learn $\mathbf{Y}$ and ${\mathbf{L}}_G$. The optimization problem becomes the following:

$$\begin{gathered} \underset{\mathbf{Y},{\mathbf{L}}_G\in ℒ}{\text{min}}\{-\text{Tr}({\mathbf{Y}}^{T}T(\mathbf{X}))+{\Vert A(\mathbf{Y})\Vert }_S\phantom{\}} \\ \phantom{\{}+\frac{\gamma }{2}\text{Tr}({\mathbf{YL}}_T{\mathbf{Y}}^T)+\frac{1}{2}\text{Tr}({\mathbf{Y}}^T{\mathbf{L}}_G\mathbf{Y})+\frac{\alpha }{2}h({\mathbf{L}}_G)\} \\ s.t.{\mathbf{Y}}^T{1}_N=0_M, \end{gathered}$$ (37)

where $\gamma$ controls the temporal smoothing effects of $T$. A similar alternating optimization algorithm can be applied for (37). Since $T$ is already fixed, solving (37) only differs from (16) in the $\mathbf{Y}$-step

$$\begin{gathered} \underset{\mathbf{Y}}{\text{min}}\{-\text{Tr}({\mathbf{Y}}^{T}T(\mathbf{X}))+{\Vert A(\mathbf{Y})\Vert }_S\phantom{\}} \\ \phantom{\{}+\frac{\gamma }{2}\text{Tr}({\mathbf{YL}}_T{\mathbf{Y}}^T)+\frac{1}{2}\text{Tr}({\mathbf{Y}}^T{\mathbf{L}}_G\mathbf{Y})\}, \\ s.t.{\mathbf{Y}}^T{1}_N=0_M, \end{gathered}$$ (38)

which uses both ${\mathbf{L}}_G$ and ${\mathbf{L}}_T$ for denoising. To use the Newton-Raphson method, we have

$$\begin{gathered} {\nabla }_{{\mathbf{y}}_j}=-T({\mathbf{x}}_j)+A^{\prime }({\mathbf{y}}_j)+{\mathbf{L}}_G{\mathbf{y}}_j+ \\ \gamma (2{\mathbf{y}}_j-{\mathbf{y}}_{j-1}-{\mathbf{y}}_{j+1}), \end{gathered}$$ (39)

$${\nabla }_{{\mathbf{y}}_j}^2=\text{diag}(A^{″}({\mathbf{y}}_j))+{\mathbf{L}}_G+2\gamma {\mathbf{I}}_N,$$ (40)

and solve Eq. (22) for the constrained updates. We term this method GLEN with Time-Vertex (GLEN-TV).

VI. Experiments

In this section, we demonstrate GLEN and its extensions on synthetic graphs as well as multiple real-world datasets. All experiments are conducted with MATLAB.

A. Synthetic Graphs

Synthetic experiments are important for evaluating graph learning methods because ground truth graphs are not frequently unavailable for real datasets. We simulate the generation of smooth graph signals using known graphs and evaluate the inferred graphs against them. We consider three random graph models: 1) Erdő s-Ré nyi model with parameter $p=0.3$; 2) Stochastic block model with two equal-sized blocks, intracommunity parameter $p=0.4$ and inter-community parameter $q=0.1$; and 3) Watts-Strogatz small-world model, where we create an initial ring lattice with node degree $2K=4$ and rewire every edge with probability $p=0.1$. Using each model, we generate 50 random graphs of $N=20$ or $N=50$ nodes, respectively. The edge weights in $\mathbf{W}$ of each graph are randomly sampled from a uniform distribution $𝒰(0.1,2)$, and the unnormalized Laplacian is computed as ${\mathbf{L}}_u=\mathbf{D}-\mathbf{W}$. We then normalize it as ${\mathbf{L}}_0=N\frac{{\mathbf{L}}_u}{\text{Tr}({\mathbf{L}}_u)}$ to obtain the ground-truth Laplacian. The signals of various data types are then generated from the process in (13). We use the Poisson distribution and the Binomial distribution to demonstrate the efficacy of GLEN. We simulate $M=2000$ unbounded and bounded count signals respectively from each distribution and infer the synthetic graphs from these generated signals. The parameter of the Binomial distribution is set to $K=8$. We use the GSPBOX [57] and the SWGT toolbox [58] for graph generation and signal processing.

We compare GLEN with leading graph Laplacian learning methods: shifted combinatorial graph Laplacian learning (SCGL) [56], two methods for graph learning from smooth signals: GLS-1 [11] and GLS-2 [12], and combinatorial graph Laplacian learning (CGL) [13]. Because existing methods are designed for Gaussian distributions, we pre-process the non-Gaussian input signals to accommodate the model mismatch. For the Poisson noise model, we first log-transform and then centralize the count signals: $\text{log}(\mathbf{x}+1_N)$. For the Binomial noise model, we apply $\text{logit}(\frac{\mathbf{x}+1}{K+2})$ where $\text{logit}(\cdot )=\text{log}\frac{1}{1-\cdot }$. These pre-processing strategies are also used to initialize the smooth representation $\mathbf{Y}$ in GLEN. For all the CGL experiments in this paper, we use $\mathbf{H}=2{\mathbf{I}}_N-1_N{1}_N^T$ to regularize $\mathbf{L}$, corresponding to type-1 regularization in [13]. For GLEN, we use type-2 regularization $\mathbf{H}={\mathbf{I}}_N-1_N{1}_N^T$.

We quantitatively compare our proposed methods with the baselines in terms of structure prediction and weight prediction. For structure prediction, we report the area under the precision-recall curve (PR-AUC), F-score, normalized mutual information (NMI) [11], and DeltaCon [59]. These metrics evaluate the binary pattern of predicted edges given by the inferred Laplacian ${\mathbf{L}}^{\ast }$, but ignore the graph weights. We obtain the precision-recall curve from a series of thresholds and calculate the F-score, NMI, and DeltaCon using the best threshold on the curve (in terms of F-score). For weight prediction, we report the relative error (RE) of estimated Laplacians, edges, and degrees against the ground truth, which reflects both structure and weight prediction. Following [13], we first normalize the inferred Laplacian ${\mathbf{L}}^{\ast }$ to obtain $\widehat{\mathbf{L}}=\frac{{\mathbf{L}}^{\ast }}{\text{Tr}({\mathbf{L}}^{\ast })}N$. The relative error of Laplacian, is then computed as ${\mathbf{RE}}_{\mathbf{L}}=\frac{{\Vert \widehat{\mathbf{L}}-{\mathbf{L}}_0\Vert }_F}{{\Vert {\mathbf{L}}_0\Vert }_F}$. For the relative error of edges, we vectorize the upper triangle of $\mathbf{W}$ to obtain $\text{vech}(\mathbf{L})\in {\mathbb{R}}^{N(N-1)∕2}$, and compute the relative ${\ell }_2$ norm ${\mathbf{RE}}_{edge}=\frac{{\Vert \text{vech}(\widehat{\mathbf{L}})-\text{vech}({\mathbf{L}}_0)\Vert }_2}{{\Vert \text{vech}({\mathbf{L}}_0)\Vert }_2}$. For the relative error of degrees, we compute the relative ${\ell }_2$ norm of the degrees ${\mathbf{RE}}_{deg}=\frac{{\Vert \text{diag}(\widehat{\mathbf{L}})-\text{diag}({\mathbf{L}}_0)\Vert }_2}{{\Vert \text{diag}({\mathbf{L}}_0)\Vert }_2}$, where $\text{diag}(\mathbf{L})$ is the vector of diagonal elements of the Laplacian.

Trade-off parameters are important for graph learning methods to impose appropriate regularization. For the selection of these hyperparameters, we perform a grid search in the parameter space of each method. We use the trade-off parameters that achieve the lowest average ${\text{RE}}_{\mathbf{L}}$ across 50 random graphs and report the evaluating metrics of the inferred graphs under these settings. We report the average values and the standard deviations for comparison.

Tab. II and III show the quantitative results of each method for the Poisson and Binomial noise models when $N=20$. Similarly, results of $N=50$ are shown in Tab. IV and V. We bold the best performance. As we can see, GLEN outperforms the baselines on all metrics for both Poisson and Binomial noise models. In particular, GLEN is substantially better than the baselines on the weight prediction metrics. This demonstrates that specifying the correct noise model is crucial for learning accurate graph geometries. The standard deviations of GLEN’s results are also smaller than the baselines in most scenarios, showing that GLEN is more stable and robust to varying graph generators. Fig. 2 compares the inferred graphs of different methods (SCGL, GLS-1, GLS-2, CGL and GLEN) to the target ground truth graphs for the random graph models.

TABLE II.

Numerical comparison of graph learning with Poisson noise (N=20).

Method	${\mathbf{RE}}_{\mathbf{L}}\downarrow$	${\mathbf{RE}}_{edge}\downarrow$	${\mathbf{RE}}_{deg}\downarrow$	PR-AUC ↑	F-score ↑	NMI ↑	DeltaCon ↑
Erdős-Rényi Model
SCGL [56]	0.226 ± 0.024	0.434 ± 0.027	0.154 ± 0.038	0.914 ± 0.019	0.870 ± 0.030	0.574 ± 0.069	0.907 ± 0.015
GLS-1 [11]	0.218 ± 0.024	0.427 ± 0.038	0.146 ± 0.032	0.845 ± 0.031	0.807 ± 0.035	0.460 ± 0.071	0.871 ± 0.018
GLS-2 [12]	0.246 ± 0.032	0.453 ± 0.045	0.179 ± 0.040	0.882 ± 0.025	0.839 ± 0.031	0.506 ± 0.073	0.891 ± 0.015
CGL [13]	0.202 ± 0.027	0.380 ± 0.031	0.142 ± 0.037	0.909 ± 0.020	0.861 ± 0.031	0.548 ± 0.073	0.900 ± 0.016
GLEN	0.129 ± 0.015	0.279 ± 0.039	0.068 ± 0.017	0.926 ± 0.018	0.890 ± 0.030	0.612 ± 0.084	0.918 ± 0.015
Stochastic Block Model
SCGL [56]	0.216 ± 0.022	0.403 ± 0.036	0.141 ± 0.027	0.930 ± 0.015	0.897 ± 0.027	0.643 ± 0.074	0.911 ± 0.016
GLS-1 [11]	0.217 ± 0.024	0.409 ± 0.039	0.140 ± 0.028	0.841 ± 0.033	0.817 ± 0.032	0.493 ± 0.071	0.870 ± 0.017
GLS-2 [12]	0.243 ± 0.026	0.432 ± 0.040	0.173 ± 0.029	0.882 ± 0.031	0.856 ± 0.053	0.569 ± 0.079	0.891 ± 0.018
CGL [13]	0.207 ± 0.022	0.370 ± 0.027	0.146 ± 0.029	0.924 ± 0.018	0.889 ± 0.028	0.633 ± 0.070	0.909 ± 0.018
GLEN	0.116 ± 0.015	0.234 ± 0.044	0.063 ± 0.013	0.941 ± 0.014	0.920 ± 0.025	0.701 ± 0.071	0.925 ± 0.015
Watts-Strogatz Model
SCGL [56]	0.120 ± 0.013	0.214 ± 0.023	0.072 ± 0.014	0.938 ± 0.020	0.930 ± 0.026	0.747 ± 0.061	0.933 ± 0.025
GLS-1 [11]	0.176 ± 0.017	0.318 ± 0.031	0.101 ± 0.019	0.878 ± 0.035	0.880 ± 0.037	0.638 ± 0.072	0.896 ± 0.019
GLS-2 [12]	0.208 ± 0.018	0.357 ± 0.030	0.137 ± 0.020	0.900 ± 0.034	0.895 ± 0.033	0.665 ± 0.066	0.896 ± 0.019
CGL [13]	0.184 ± 0.016	0.306 ± 0.023	0.129 ± 0.019	0.950 ± 0.015	0.941 ± 0.024	0.769 ± 0.063	0.806 ± 0.015
GLEN	0.100 ± 0.012	0.172 ± 0.019	0.065 ± 0.013	0.960 ± 0.011	0.960 ± 0.019	0.813 ± 0.055	0.941 ± 0.015

TABLE III.

Numerical comparison of graph learning with Binomial noises (N=20).

Method	${\mathbf{RE}}_{\mathbf{L}}\downarrow$	${\mathbf{RE}}_{edge}\downarrow$	${\mathbf{RE}}_{deg}\downarrow$	PR-AUC ↑	F-score ↑	NMI ↑	DeltaCon ↑
Erdős-Rényi Model
SCGL [56]	0.161 ± 0.020	0.342 ± 0.054	0.089 ± 0.022	0.904 ± 0.017	0.868 ± 0.029	0.575 ± 0.077	0.893 ± 0.025
GLS-1 [11]	0.202 ± 0.027	0.395 ± 0.035	0.133 ± 0.029	0.868 ± 0.024	0.840 ± 0.027	0.511 ± 0.064	0.887 ± 0.019
GLS-2 [12]	0.277 ± 0.030	0.449 ± 0.035	0.228 ± 0.030	0.891 ± 0.018	0.842 ± 0.024	0.513 ± 0.056	0.893 ± 0.014
CGL [13]	0.220 ± 0.028	0.338 ± 0.036	0.188 ± 0.030	0.914 ± 0.014	0.877 ± 0.025	0.593 ± 0.067	0.909 ± 0.014
GLEN	0.112 ± 0.012	0.234 ± 0.033	0.064 ± 0.014	0.935 ± 0.015	0.901 ± 0.027	0.641 ± 0.083	0.921 ± 0.015
Stochastic Block Model
SCGL [56]	0.171 ± 0.027	0.301 ± 0.038	0.121 ± 0.033	0.920 ± 0.021	0.895 ± 0.028	0.651 ± 0.071	0.906 ± 0.019
GLS-1 [11]	0.236 ± 0.037	0.412 ± 0.048	0.170 ± 0.041	0.859 ± 0.036	0.848 ± 0.037	0.556 ± 0.077	0.882 ± 0.021
GLS-2 [12]	0.302 ± 0.040	0.460 ± 0.047	0.251 ± 0.042	0.884 ± 0.030	0.846 ± 0.033	0.546 ± 0.068	0.884 ± 0.019
CGL [13]	0.229 ± 0.031	0.330 ± 0.038	0.198 ± 0.032	0.923 ± 0.019	0.896 ± 0.026	0.651 ± 0.062	0.911 ± 0.017
GLEN	0.112 ± 0.019	0.207 ± 0.036	0.074 ± 0.018	0.945 ± 0.015	0.929 ± 0.025	0.726 ± 0.067	0.929 ± 0.016
Watts-Strogatz Model
SCGL [56]	0.149 ± 0.018	0.229 ± 0.024	0.115 ± 0.019	0.951 ± 0.019	0.945 ± 0.028	0.779 ± 0.072	0.928 ± 0.023
GLS-1 [11]	0.213 ± 0.023	0.347 ± 0.033	0.152 ± 0.026	0.884 ± 0.034	0.895 ± 0.034	0.667 ± 0.070	0.895 ± 0.021
GLS-2 [12]	0.241 ± 0.028	0.372 ± 0.035	0.186 ± 0.030	0.919 ± 0.028	0.902 ± 0.032	0.673 ± 0.071	0.898 ± 0.020
CGL [13]	0.187 ± 0.022	0.284 ± 0.021	0.145 ± 0.028	0.955 ± 0.014	0.946 ± 0.024	0.782 ± 0.062	0.928 ± 0.018
GLEN	0.098 ± 0.012	0.161 ± 0.018	0.069 ± 0.013	0.964 ± 0.009	0.964 ± 0.022	0.827 ± 0.069	0.943 ± 0.018

TABLE IV.

Numerical comparison of graph learning with Poisson noise (N=50).

Method	${\mathbf{RE}}_{\mathbf{L}}\downarrow$	${\mathbf{RE}}_{edge}\downarrow$	${\mathbf{RE}}_{deg}\downarrow$	PR-AUC ↑	F-score ↑	NMI ↑	DeltaCon ↑
Erdős-Rényi Model
SCGL [56]	0.263 ± 0.025	0.774 ± 0.062	0.161 ± 0.026	0.741 ± 0.017	0.672 ± 0.017	0.228 ± 0.029	0.776 ± 0.009
GLS-1 [11]	0.192 ± 0.010	0.614 ± 0.019	0.095 ± 0.016	0.754 ± 0.015	0.678 ± 0.017	0.230 ± 0.030	0.782 ± 0.011
GLS-2 [12]	0.199 ± 0.009	0.612 ± 0.022	0.111 ± 0.013	0.760 ± 0.015	0.683 ± 0.016	0.236 ± 0.031	0.791 ± 0.009
CGL [13]	0.206 ± 0.008	0.613 ± 0.021	0.122 ± 0.013	0.754 ± 0.016	0.678 ± 0.018	0.230 ± 0.031	0.791 ± 0.008
GLEN	0.177 ± 0.006	0.610 ± 0.030	0.060 ± 0.008	0.767 ± 0.015	0.690 ± 0.017	0.248 ± 0.031	0.800 ± 0.007
Stochastic Block Model
SCGL [56]	0.213 ± 0.016	0.608 ± 0.046	0.117 ± 0.016	0.789 ± 0.021	0.713 ± 0.022	0.324 ± 0.038	0.796 ± 0.011
GLS-1 [11]	0.195 ± 0.009	0.566 ± 0.019	0.101 ± 0.015	0.796 ± 0.020	0.717 ± 0.021	0.327 ± 0.033	0.798 ± 0.011
GLS-2 [12]	0.202 ± 0.009	0.555 ± 0.022	0.121 ± 0.012	0.801 ± 0.018	0.726 ± 0.020	0.342 ± 0.033	0.808 ± 0.009
CGL [13]	0.210 ± 0.009	0.556 ± 0.021	0.134 ± 0.012	0.800 ± 0.020	0.718 ± 0.021	0.331 ± 0.038	0.806 ± 0.010
GLEN	0.169 ± 0.007	0.523 ± 0.028	0.066 ± 0.010	0.811 ± 0.019	0.734 ± 0.021	0.352 ± 0.037	0.815 ± 0.009
Watts-Strogatz Model
SCGL [56]	0.147 ± 0.014	0.249 ± 0.021	0.099 ± 0.014	0.963 ± 0.010	0.935 ± 0.016	0.813 ± 0.032	0.908 ± 0.014
GLS-1 [11]	0.192 ± 0.015	0.337 ± 0.024	0.120 ± 0.016	0.856 ± 0.028	0.853 ± 0.024	0.664 ± 0.040	0.866 ± 0.015
GLS-2 [12]	0.220 ± 0.017	0.367 ± 0.025	0.152 ± 0.018	0.886 ± 0.023	0.874 ± 0.020	0.697 ± 0.036	0.866 ± 0.014
CGL [13]	0.196 ± 0.016	0.319 ± 0.020	0.140 ± 0.019	0.965 ± 0.009	0.935 ± 0.015	0.814 ± 0.032	0.904 ± 0.016
GLEN	0.110 ± 0.014	0.180 ± 0.021	0.078 ± 0.014	0.975 ± 0.008	0.967 ± 0.013	0.884 ± 0.032	0.922 ± 0.012

TABLE V.

Numerical comparison of graph learning with Binomial noises (N=50).

Method	${\mathbf{RE}}_{\mathbf{L}}\downarrow$	${\mathbf{RE}}_{edge}\downarrow$	${\mathbf{RE}}_{deg}\downarrow$	PR-AUC ↑	F-score ↑	NMI ↑	DeltaCon ↑
Erdős-Rényi Model
SCGL [56]	0.193 ± 0.010	0.647 ± 0.041	0.079 ± 0.009	0.803 ± 0.015	0.719 ± 0.017	0.291 ± 0.034	0.804 ± 0.007
GLS-1 [11]	0.168 ± 0.005	0.547 ± 0.024	0.077 ± 0.008	0.799 ± 0.014	0.718 ± 0.017	0.289 ± 0.031	0.807 ± 0.010
GLS-2 [12]	0.212 ± 0.010	0.560 ± 0.021	0.152 ± 0.015	0.800 ± 0.016	0.716 ± 0.017	0.289 ± 0.031	0.807 ± 0.007
CGL [13]	0.229 ± 0.016	0.558 ± 0.024	0.177 ± 0.020	0.795 ± 0.017	0.709 ± 0.017	0.282 ± 0.034	0.803 ± 0.007
GLEN	0.159 ± 0.005	0.540 ± 0.026	0.061 ± 0.006	0.804 ± 0.016	0.720 ± 0.016	0.294 ± 0.032	0.811 ± 0.008
Stochastic Block Model
SCGL [56]	0.165 ± 0.007	0.514 ± 0.032	0.064 ± 0.008	0.846 ± 0.016	0.764 ± 0.021	0.402 ± 0.035	0.821 ± 0.008
GLS-1 [11]	0.165 ± 0.007	0.484 ± 0.022	0.083 ± 0.011	0.833 ± 0.016	0.760 ± 0.019	0.398 ± 0.032	0.819 ± 0.008
GLS-2 [12]	0.215 ± 0.013	0.507 ± 0.023	0.160 ± 0.016	0.839 ± 0.015	0.760 ± 0.017	0.398 ± 0.034	0.817 ± 0.007
CGL [13]	0.224 ± 0.013	0.503 ± 0.020	0.174 ± 0.016	0.831 ± 0.017	0.749 ± 0.019	0.381 ± 0.039	0.815 ± 0.009
GLEN	0.150 ± 0.006	0.465 ± 0.026	0.059 ± 0.009	0.843 ± 0.016	0.765 ± 0.020	0.405 ± 0.038	0.826 ± 0.008
Watts-Strogatz Model
SCGL [56]	0.193 ± 0.015	0.290 ± 0.020	0.153 ± 0.015	0.969 ± 0.009	0.943 ± 0.015	0.829 ± 0.034	0.905 ± 0.014
GLS-1 [11]	0.228 ± 0.016	0.350 ± 0.019	0.177 ± 0.020	0.878 ± 0.025	0.876 ± 0.023	0.699 ± 0.041	0.861 ± 0.015
GLS-2 [12]	0.252 ± 0.020	0.380 ± 0.025	0.200 ± 0.022	0.901 ± 0.023	0.885 ± 0.021	0.715 ± 0.037	0.864 ± 0.016
CGL [13]	0.219 ± 0.013	0.351 ± 0.018	0.161 ± 0.017	0.967 ± 0.009	0.937 ± 0.016	0.817 ± 0.034	0.902 ± 0.015
GLEN	0.116 ± 0.015	0.177 ± 0.016	0.092 ± 0.017	0.976 ± 0.006	0.959 ± 0.013	0.865 ± 0.032	0.925 ± 0.017

Fig. 2.

Examples of the inferred and ground truth graph Laplacian. The synthetic signals are generated with the Binomial noise model. Rows correspond to different graph generators and columns correspond to different graph inference methods or ground truth graphs.

B. Chicago Crime Dataset

Now we evaluate GLEN on a real dataset, the Chicago Crime Dataset [60]. Our goal is to learn a graph between different types of crime to reveal their patterns of concurrence. The dataset contains 32 types of crimes that occurred in 77 Chicago communities during every hour from 2001 to 2017. We first bin the data by year over the last 10 years and sum the counts within each bin, which leads to 770 graph signals over 32 crime types. We further remove “Ritualism” from the crime list since there is no record in the 10 years. This results in a 31 × 770 count matrix, and we apply the graph learning methods to it.

Since each matrix entry indicates the number of crime occurrences in a specific region during one of the 10 years, we use Poisson distribution to model the count signals. However, since the crime rates vary dramatically by the type of crimes, we use the extended GLEN with offset variables as in Sec. V-A where the Poisson distribution is used. The offset of each crime is initialized as the logarithm of their mean occurrences. We compare our method with CGL and tune their hyperparameters to encourage sparsity. The crime graphs learned by both methods are shown in Fig. 3. As we can see, CGL is more biased by the number of crime occurrences, tending to link crimes with similar frequencies. Furthermore, it fails to converge when we increase the regularization term to estimate a sparser graph. On the other hand, GLEN learns a more interpretable graph that better links similar crimes. For example, GLEN learns an edge between “Sex Offense”, “Criminal Sexual Assault”, and “Assualt”, which is ignored by CGL. The graph learned by GLEN is also more agnostic to the crime frequencies.

Fig. 3.

Rank of the crimes by their mean occurrences (a), and graphs inferred from the Chicago crime dataset using original CGL (b) and GLEN with offsets (c). Nodes correspond to crime types. The width of the edges corresponds to the edge weights.

C. Animals Dataset

We also evaluate our method on the Animals dataset [61]. The Animal dataset is a binary matrix of size 33 × 102, where each row corresponds to an animal species and each column corresponds to a boolean feature such as “has wings?”, “has lungs?”, “is dangerous?”. Our goal is to learn a graph where each node represents a species and each edge represents the similarity between them. To accommodate smooth models to the binary signals, previous work [13], [14] used a heuristic statistic $\mathbf{S}=\frac{1}{M}{\mathbf{XX}}^T+\frac{1}{3}\mathbf{I}$ suggested by [62]. We instead explicitly model the binary signals using the Bernoulli distribution, resulting in an improper Bernoulli-Logit-Normal model [54]. This amounts to setting $R=1$ in the Binomial noise model for GLEN.

We evaluate GLEN against existing methods both qualitatively and quantitatively. Although a ground-truth animal graph does not exist, we construct a pseudo-ground-truth using the scientific animal classification. Animal classification is a taxonomy that categorizes living forms into an 8-level hierarchical system: ‘domains’, ‘kingdoms’, ‘phyla’, ‘classes’, ‘orders’, ‘families’, ‘genera’, and ‘species’. For example, both tiger and squirrel belong to the Mammalia ‘class’, but they belong to Carnivora and Rodentia, respectively, on the more specific ‘order’ level. Their highest common level indicates their similarity. Thus, for a set of nodes that correspond to the listed animals, we connect two nodes if they belong to the same ‘class’. Given that over half of the animals belong to the Mammalia ‘class’, we only connect two Mammalia nodes if they also belong to the same ‘order’. This gives us an unweighted graph that respects the animal similarities.

Fig. VI-C shows the weighted adjacency matrices learned by GLS-1, CGL, and GLEN. First, we can see that GLEN learns a sparser graph with more clear community structures, compared with GLS-1 and CGL. GLEN identifies clear community structures between animal ‘classes’ and Mammalia ‘orders’. In contrast to GLEN, CGL ignores the insect subnetwork “Bee-Butterfly-Ant-Cockroach” while GLS-1 learns more spurious edges. Next, we quantify the accuracy of the learned graphs using the ground-truth taxonomy graph. We perform a grid search for each method and report their highest PR-AUC. As shown in Table VI, GLEN achieves the highest PR-AUC. This implies that the graph learned by GLEN best reflects the animal taxonomy.

TABLE VI.

Comparing PR-AUC on the Animal Classification.

Method	GLS-1	GLS-2	CGL	GLEN
PR-AUC	81.50	80.39	68.02	82.63

D. Neural Dataset

We now turn to a dataset that has a graph-temporal structure. The Area2_Bump dataset from the neural latent benchmark (NLB) [63] consists of multiple trials of neural spiking activity and simultaneous behavior data of a macaque. The macaque is trained to control a cursor and perform center-out reaches toward one of eight target directions. In a subset of random trials, a bump interrupts the macaque shortly before the reach. The macaque’s neural activity is recorded from Brodmann’s area 2 of the somatosensory cortex, which has been shown to contain information about whole-arm kinematics. The neural recording is contained in a non-negative integer matrix $\mathbf{X}\in {\mathbb{Z}}_{+}^{N\times T\times K}$, where each entry ${\mathbf{X}}_{itk}$ counts the firing of neuron $i\in \{1,2,\dots ,N\}$ in time bin $t\in \{1,2,\dots ,T\}$ during trial $k\in \{1,2,\dots ,K\}$. Following the standard procedure in [63], we resample the 1-ms resolution signals into 5-ms bins.

Learning the interactions between the neurons from these spiking activity matrices relates functional connectivity patterns to behavior. Suppose a graph underlies the set of $N$ neurons, a graph signal ${\mathbf{X}}_{:tk}$ is the simultaneous spiking of all neurons at time $t$ in trial $k$. Since graph signals are not independent but temporally correlated, we use GLEN-TV instead of GLEN to denoise the signals spatially and temporally. We infer a graph of neurons for each trial, bumped or not bumped, and analyze the inferred graphs across different conditions, the target directions. We model the observation with the Poisson distribution which is used in the standardized co-smoothing evaluation [64]. The objective function and alternating updates are obtained by plugging in the Poisson distribution to Eq. (37).

We perform linear discriminant analysis (LDA) on the degree vector of learned Laplacians, using direction conditions as the eight class labels. LDA achieves 70.33% accuracy, indicating that the structural information in our learned graphs encodes the class conditions. When applying GLEN without the temporal modeling, LDA only achieves 67.86% accuracy, which indicates the importance of modeling temporal correlations. We also visualize the average spiking activity and the averaged denoised signals for all 8 conditions in Fig. 5. The denoised signals are the exponential of learn smooth representation $\mathbf{Y}$, which are interpreted as the firing rates of the neurons. Note that GLEN-TV smooths the signals both spatially and temporally.

Fig. 5.

Average raw neural spiking data compared against average denoised signals by GLEN-TV, for each target direction shown on the left. The brighter color indicates a higher firing rate.

VII. Conclusions

In this paper, we generalize combinatorial graph Laplacian learning to account for various non-Gaussian data types. Inspired by generalized linear models, we established a GSP-based generative model that produces signals of different responses of exponential family distributions whose expected values are underlying smooth signal representations. To effectively learn the graph Laplacian, we proposed GLEN which alternates between a graph learning step that refines the estimation of the Laplacian and a denoising step that refines the underlying smooth signal representations until convergence. Since real-world graph signals are often not independent but temporally correlated, we extended GLEN to the time-vertex framework for learning from time-varying graph signals. We demonstrated that our methods achieve more accurate Laplacian estimation on different synthetic graph models compared to competing approaches, and can learn meaningful graphs on various real datasets. While our experiments highlighted Poisson and Binomial distributions, our framework is general for exponential family distributions.

This paper serves as a first step toward a GSP model of multivariate non-Gaussian distributions and opens multiple directions for future works. One direction is the theoretical aspects of GLEN. Studying its consistency properties as a function of $N$ and $M$ can help us better understand its behavior. Quantifying the error and variance of the estimator under assorted graph similarity measures [65], [66] is also intriguing. Another interesting direction is to study the MLE of (13) since GLEN is not exact. The exact MLE is intractable for general exponential family distributions, but the variational and Monte Carlo methods might be used to approximate the intractable likelihood. Finally, on the more practical side, the exponential family noise model (13) can be further generalized. From the perspective of noise distributions, although we dramatically enlarged the scope of data types of the GSP-based graph inference methods, our model can be more versatile by encompassing dispersion or Bayesian inference. From the graph learning perspective, learning multiple graphs is an interesting problem featured by recent works [67], [68]. Generalizing our model to multi-dimensional tensors is also of great interest [69].

Fig. 4.

Heat-maps of the inferred graphs of GLS-1, GLS-2, CGL, and GLEN. Animals are ordered to reflect their taxonomy, where each red square highlights one of the 5 animal ‘classes’. The corresponding animal ‘classes’ and the ‘orders’ of mammals are shown on the right.