Model-Based Clustering of Nonparametric Weighted Networks with Application to Water Pollution Analysis

Abstract

Water pollution is a major global environmental problem, and it poses a great environmental risk to public health and biological diversity. This work is motivated by assessing the potential environmental threat of coal mining through increased sulfate concentrations in river networks, which do not belong to any simple parametric distribution. However, existing network models mainly focus on binary or discrete networks and weighted networks with known parametric weight distributions. We propose a principled nonparametric weighted network model based on exponential-family random graph models and local likelihood estimation, and study its model-based clustering with application to large-scale water pollution network analysis. We do not require any parametric distribution assumption on network weights. The proposed method greatly extends the methodology and applicability of statistical network models. Furthermore, it is scalable to large and complex networks in large-scale environmental studies. The power of our proposed methods is demonstrated in simulation studies and a real application to sulfate pollution network analysis in Ohio watershed located in Pennsylvania, United States.

1. Introduction

Water pollution is the leading cause of deaths and diseases, and it is a major global problem. It is known that nearly 80% of the world’s population lives in areas exposed to high levels of threat to water security (Vörösmarty et al., 2010). The recent national report on water quality by EPA (2017) pointed out that 46% of rivers, 21% of lakes, 18% of coastal waters, and 32% of wetlands in the United States are in poor biological condition or rated poor based on a water quality index. The major pollutant sources include agriculture, atmospheric deposition, construction, industrial production, municipal sewage, resource extraction, spills, and urban runoff. They pose severe health hazards like cancer, cardiovascular, respiratory, neurologic, and developmental damage. This work is motivated by assessing the potential environmental threat of coal mining in Ohio watershed of Pennsylvania through increased sulfate concentrations in the surface water, which is an important scientific problem in geoscience. Bernhardt et al. (2012) mapped surface coal mining of southern West Virginia and linked these maps with water quality and biological data of 223 streams. When these coal mines occupy > 5.4% of their contributing watershed area, the sulfate concentrations within catchments could exceed 50 mg/L (Niu et al., 2017). Residential proximity to heavy coal production is associated with higher risk for cardiopulmonary disease, chronic lung disease, hypertension, and kidney disease (Hendryx and Ahern, 2008). The study of water-quality risks will help the whole society manage them now and in future.

With advances in data collection, there are more and more modern statistical research on environmental studies using nonparametric regression, causal inference, mixture model, network analysis, and variable selection, for instance, Ebenstein (2012); Liang et al. (2015); Li et al. (2017); Lin et al. (2017); Wen et al. (2019) among many others. Especially, network analysis becomes increasingly important in large-scale environmental studies and geoscientific research to assess environmental impacts and risks for water pollution (Smith et al., 1987; Lienert et al., 2013; Ruzol et al., 2017). For example,, Gianessi and Peskin (1981) proposed a water network model to explore the impact of cropland sediment controls on improved water quality, and Montgomery (1972) and Anastasiadis et al. (2016) studies weighted pollution networks where the weights measure the pollution diminishing transition. However, none of aforementioned network models took into account the spatial heterogeneity and the hub structure of river networks. Without exploring spatial heterogeneity, these models could fail to differentiate polluted regions from less polluted but well connected regions in river networks. The hubs in river networks usually determine the flow of pollutants, and they may help identify polluted and well connected regions.

In this work, we introduce a principled model-based clustering of networks to effectively deal with the spatial heterogeneity of river pollution and efficiently identify the hub structure in river networks. Model-based clustering of networks based on stochastic block models (SBMs) and exponential-family random graph models (ERGMs) have received considerable attention in recent literature, including Snijders and Nowicki (1997); Nowicki and Snijders (2001); Girvan and Newman (2002); Airoldi et al. (2008); Karrer and Newman (2011); Zhao et al. (2012); Vu et al. (2013); Saldana et al. (2017); Wang and Bickel (2017); Lee et al. (2017), among many others. It is worth pointing out that existing research mainly focuses on the model-based clustering of networks with binary or discrete edges. In a recent paper by Ambroise and Matias (2012), parametric distributions are incorporated into a stochastic block model to model continuous network weights. Alternatively, Bayesian variational methods have been proposed by Aicher et al. (2014) to approximate posterior distribution of weights over latent block structures. However, from our motivating example, sulfate concentrations in the surface river network do not belong to any simple parametric distribution, which will be illustrated in Section 5. Thus, we need to relax the parametric assumption in network models to account for the unknown distribution of continuous network weights. To address this issue, we propose a new nonparametric weighted network model based on ERGMs and local likelihood estimation, and study its model-based clustering with application to large-scale water pollution network analysis. The proposed method greatly extends the methodology and applicability of statistical network models. Furthermore, it is scalable to large and complex networks in real-world applications.

The rest of this paper is organized as follows. Section 2 presents the methodology of our proposed nonparametric weighted network model. In Section 3, we introduce a novel variational expectation-maximization algorithm to solve the approximate maximum likelihood estimation. Section 4 demonstrates the numerical performances of our proposed methods and algorithms in simulation studies. In Section 5, we apply the proposed method to analyze the large-scale water pollution network of sulfate concentrations in the Ohio watershed of Pennsylvania.

2. Methodology

We define some necessary notation before presenting our proposed method. Let n be the number of nodes in the observed network. Let Y = (Y_ij )_{1≤i, j≤n} be the corresponding weighted network such that Y_ij = (E_ij,W_ij) where E_ij is a binary indicator denoting the existence of an edge in the network and W_ij is the corresponding weight when E_ij = 1. The weight matrix W = (W_ij)_1≤i,j≤n, consists of continuous weights in the network. Further we assume that the edge distribution of each network belongs to an exponential family (Besag, 1974; Frank and Strauss, 1986). We write the distribution of edge indicator matrix E as

$$P_{\mathit{\theta }}(\mathit{E}=\mathit{e})=\exp \left\{{\mathit{\theta }}^{\prime }\mathit{g}(\mathit{e})-\psi (\mathit{\theta })\right\},$$ (1)

where $\psi (\mathit{\theta })=\log \sum _{e\in \mathcal{E}}\exp \left[{\mathit{\theta }}^{\prime }\mathit{g}(\mathit{e})\right]$ is the log of the normalizing constant, θ ∈ ℝ^p are canonical network parameters of interest and $\mathit{g}:\mathcal{E}\to {ℝ}^p$ is the sufficient statistic. Here $\mathcal{E}$ is the space for E consisting of 2ⁿ possible binary edge structures.

One of the major limitations of this binary network model is that it can not deal with large number of nodes due to large computational time for evaluating the likelihood function. For undirected networks, this computing time scales with node size as exp((n(n−1)log2)/2). Many estimation algorithms have been developed (Snijders, 2002; Hunter and Handcock, 2006; Møller et al., 2006; Koskinen et al., 2010; Caimo and Friel, 2011), however most of them are time-consuming and therefore unrealistic for fitting large networks. This issue of non-scalability can be resolved by the assumption of dyadic independence which assumes that all dyads are independent of each other. Note that dyad is a general term applicable for both directed and undirected edges. In the undirected weighted networks, it would imply the ties and weights are independent of each other and for all pairs of nodes, i.e.

$$P_{\mathit{\theta }}(\mathit{Y}=\mathit{y})=\prod _{1\le i<j\le n}\exp \left\{{\mathit{\theta }}^{\prime }\mathit{g}\left(e_{ij}\right)-\psi (\mathit{\theta })\right\}{P}_{\mathit{\theta }}\left(W_{ij}=w_{ij}\right),$$ (2)

This assumption facilitates both estimation and simulation of large networks as well as solves the issue of degeneracy (Strauss, 1986; Handcock et al., 2003; Schweinberger, 2011; Krivitsky, 2012). However, dyadic independence is too restrictive and most models following this assumption are either very trivial, failing to capture relational dependencies (Gilbert, 1959; Erdős and Rényi, 1959) or non-parsimonious, with a large number of parameters (Holland and Leinhardt, 1981).

We consider a model-based clustering framework to relax the dyadic independence. More specifically, we introduce the finite K-component mixture form together with a much less restrictive assumption of conditional dyadic independence (CDI) (Snijders and Nowicki, 1997; Nowicki and Snijders, 2001; Girvan and Newman, 2002; Vu et al., 2013). Under this assumption, we propose the nonparametric weighted network as

$$P_{\mathit{\theta },\mathit{f}}(\mathbf{Y}=\mathbf{y}|\mathbf{Z}=\mathbf{z})=\prod _{1\le i<j\le n}{P}_{{\mathit{\theta }}_{z_i{z}_j},f_{z_i{z}_j}}(Y_{ij}=y_{ij}|\mathit{Z}=\mathit{z})$$ (3)

where Z = (Z_i)_1≤i≤n denote the latent cluster memberships of nodes. Here, Z_i is a K × 1 vector such that z_ik = 1 if and only if node i lies in cluster k, otherwise z_ik = 0. The conditional dyadic independence strikes in a nice balance between model complexity and parsimony for model-based clustering. The number of parameters are reduced from $\mathcal{O}\left(n^2\right)$ to $\mathcal{O}\left(K^2\right)$, thus enabling simple inter and intra community interpretations. The CDI assumption induces a block structure similar to SBMs. SBMs have been thoroughly studied in the context of social networks (Holland et al., 1983; Airoldi et al., 2008) and have a long history in multiple scientific communities (Bui et al., 1987; Dyer and Frieze, 1989; Bollobás et al., 2007).

We omit discussion of SBMs here except to point out a major difference from our current setup of ERGM. ERGMs can allow several kinds of dynamic network statistics like density, stability, transitivity (Hanneke et al., 2010), thus effectively generalizing the simple density case in SBM methodology which makes them appealing in practice. To effectively model continuous weights, for any given pair of nodes (i, j), we have

$$P_{\mathit{\theta },\mathit{f}}\left(Y_{ij}=y_{ij}|\mathit{Z}=\mathit{z}\right)={\left(p_{z_i{z}_j}{f}_{z_i{z}_j}\left(w_{ij}\right)\right)}^{1_{e_{ij\ne 0}}}{\left(1-p_{z_i{z}_j}\right)}^{1_{e_{ij=0}}}$$ (4)

Here, ${\left(p_{kl}\right)}_{1\le k,l\le K}={(P_{{\mathit{\theta }}_{z_i{z}_j}}\left(E_{ij}=e_{ij}|\mathbf{Z}=\mathbf{z}\right))}_{1\le k,l\le K}$ take the parametric specification of exponential-family distributions of the network statistics as explained in (1) and network weights ${\left(w_{ij}\right)}_{1\le i,j\le n:z_i=k,z_j=l}$ are assumed to be an i.i.d. sample observed from a population following an univariate nonparametric density function f_kl. Note that (f_kl)_1≤k,l≤K do not necessarily have any parametric form. We implicitly assume that conditioning on the full Z = z is same as conditioning on just z_i and z_j, i.e. E_ij and W_ij depend on Z only via z_i and z_j. For ease of presentation, we assume the additive structure of p_kl to be parameterized by network sparsity parameters θ as p_kl = logit⁻¹(θ_k + θ_l ). It is worth pointing out that the clusters z₁,⋯,z_n are determined by two sources of information in network models. On the one hand, the random block structure and also the additive structure of (p_kl)_1≤k,l≤K contribute to the exploration of different degrees among the clusters. On the other hand, the nonparametric network weights modulate the separation between clusters with different weight distributions.

Combining (3) and (4), the corresponding log-likelihood function given the cluster memberships can be written as:

$$\begin{gathered} \log \left(P_{\mathit{\theta },\mathit{f}}(\mathbf{Y}=\mathbf{y}|\mathbf{Z}=\mathbf{z})\right) \\ =\sum _{i=1}^{n-1}\sum _{j=i+1}^n\{\left[1_{e_{ij}\ne 0}\log \left(p_{z_i{z}_j}\right)+1_{e_{ij}=0}\log \left(1-p_{z_i{z}_j}\right)\right] \\ +1_{e_{ij}\ne 0}\left[\log \left(f_{z_i{z}_j}\left(w_{ij}\right)\right)-\left({\int }_{\mathcal{X}}{f}_{z_i{z}_j}(u)du-1\right)\right]\}, \end{gathered}$$ (5)

where we also introduce the penalty term $\left({\int }_{\mathcal{X}}{f}_{kl}(u)du-1\right)$. Thus, (5) can be treated as a likelihood for any non-negative function f_kl without imposing the additional constraint ${\int }_{\mathcal{X}}{f}_{kl}(u)du=1$. This specification follows the similar spirit of Loader (1996).

Now we derive the localized version of the conditional log-likelihood evaluated at an arbirary grid point w as:

$$\begin{gathered} \ell (\mathit{\theta },\mathit{f},w;\mathit{Y}|\mathit{Z}) \\ =\sum _{i=1}^{n-1}\sum _{j=i+1}^n\{\left[1_{e_{ij}\ne 0}\log \left(p_{z_i{z}_j}\right)+1_{e_{ij}=0}\log \left(1-p_{z_i{z}_j}\right)\right]+1_{e_{ij}\ne 0}\times \\ \left[K_h\left(w_{ij}-w\right)\log \left(f_{z_i{z}_j}\left(w_{ij}\right)\right)-\left({\int }_{\mathcal{X}}{K}_h(u-w)f_{z_i{z}_j}(u)du-1\right)\right]\} \end{gathered}$$ (6)

where K_h is the rescaled kernel function with a positive bandwidth h. We approximate log(f_kl(u)) by ${\mathrm{\Phi }}_{kl}^p$, a linear combination of orthogonal basis functions (ϕ_m)_1≤m≤p, namely, ${\mathrm{\Phi }}_{kl}^p(u-w)=\sum _{m=0}^p{\beta }_m^{(kl)}{\varphi }_m(u-w)$. With this approximation, the local conditional log-likelihood becomes

$$\begin{gathered} \ell (\mathit{\theta },\mathit{\beta },w;\mathit{Y}|\mathit{Z}) \\ =\sum _{i=1}^{n-1}\sum _{j=i+1}^n\{\left[1_{e_{ij}\ne 0}\log \left(p_{z_i{z}_j}\right)+1_{e_{ij}=0}\log \left(1-p_{z_i{z}_j}\right)\right]+1_{e_{ij}\ne 0}\times \\ \left[K_h\left(w_{ij}-w\right){\mathrm{\Phi }}_{z_i{z}_j}^p\left(w_{ij}-w\right)-\left({\int }_{\mathcal{X}}{K}_h(u-w)\exp \left({\mathrm{\Phi }}_{z_i{z}_j}^p(u-w)\right)du-1\right)\right]\} \end{gathered}$$ (7)

We assume that membership indicators Z = (Z_i)_1≤i≤n follow a multinomial distribution with a single trial and mixture proportions as π = (π_k)_1≤k≤K. The log-likelihood of the observed weighted network can be written as

$$\ell (\mathit{\theta },\mathit{\pi },\mathit{f})=\log \left(\sum _{\mathbf{z}\in {\{1,\dots ,K\}}^n}{P}_{\mathit{\theta },\mathit{f}}(\mathbf{Y}=\mathbf{y}|\mathbf{Z}=\mathbf{z})P_{\pi }(\mathbf{Z}=\mathbf{z})\right)$$ (8)

In view of (5) and (7), we maximize the log-likelihood function (8) of the observed network described to estimate model parameters θ together with block densities f.

Remark 1. Allman et al. (2011) proved the identifiability of parameters for a broad class of parametric or nonparametric weighted network models. As shown in Section 4 of Allman et al. (2011), we can uniquely identify the parameters under mild conditions for parametric weighted networks while the identifiability result for nonparametric weighted networks depend on binning the values of the edge variables into a finite set. Please see Theorem 15 of Allman et al. (2011) for more details about the identifiability.

Remark 2. The proposed nonparametric weighted network model can be further extended to discrete temporally evolving weighted networks. Given the dynamic network series Y = (Y_t)_1≤t≤T = (E_t, W_t)_1≤t≤T, the dynamic nonparametric weighted network model can be derived by assuming a discrete-time Markov structure over the time (Hanneke et al., 2010; Krivitsky and Handcock, 2014; Kim et al., 2018), that is

$$P_{\mathit{\theta },\mathit{f}}(\mathbf{Y}=\mathbf{y}|\mathbf{Z}=\mathbf{z})=\prod _{2\le t\le T}{P}_{\mathit{\theta },\mathit{f}}\left({\mathbf{Y}}_t={\mathbf{y}}_t|{\mathbf{Y}}_{t-1}={\mathbf{y}}_{t-1},\mathbf{Z}=\mathbf{z}\right)$$

where P_θ,f (Y_t = y_t | Y_t−1 = y_t−1, Z = z) follows the similar specification of (3) and (4). We can incorporate dynamic network statistics such as stability and transitivity in the exponential-family specification of $P_{{\mathit{\theta }}_{z_i{z}_j}}\left(E_{t,ij}=e_{t,ij}|E_{t-1,ij}=e_{t-1,ij},\mathbf{Z}=\mathbf{z}\right)$. Hence, the interpretation of clusters will reflect the impact from dynamic network statistics. For example, the use of stability network statistic will contribute to the exploration of different levels of stability among the clusters. The dynamic nonparametric weighted network is beyond our current scope and we will study it in the future.

3. Computation

This section proposes a variational expectation maximization (EM) algorithm to approximately solve the maximum likelihood estimation. It is infeasible to directly maximize the log-likelihood function (8) due to two key challenges: (i) exponential-family form of P(Y | Z) is not scalable for large networks; (ii) the sum is over every possible assignment to Z, where for each 1≤ i ≤ n, Z_i = z_i can take one of K possible values.

To resolve the first challenge, the CDI assumption (3) plays a crucial role. Typically parameters in a mixture model are estimated using the classical EM algorithm (Dempster et al., 1977). The E-step proceeds by writing the complete data log-likelihood (9), assuming the network is observed while node membership indicators Z are unobserved.

$$\begin{gathered} \log \left(P_{\mathit{\theta },\mathit{\pi },\mathit{f}}(\mathbf{Y}=\mathbf{y},\mathbf{Z}=\mathbf{z})\right) \\ =\sum _{i=1}^{n-1}\sum _{j=i+1}^n\sum _{k=1}^K\sum _{l=1}^K{z}_{ik}{z}_{jl}\log P_{\mathbf{\theta }}\left(Y_{ij}=y_{ij}|\mathbf{Z}=\mathbf{z}\right)+\sum _{i=1}^n\sum _{k=1}^K{z}_{ik}\log \left({\pi }_k\right) \end{gathered}$$ (9)

Next we take expectation of this complete log-likelihood with the distribution ℙ_θ(Z | Y). Clearly the distribution ℙ_θ(Z | Y) = ℙ_θ(Z_i,1≤i≤n | Y) can’t be further factored over nodes since for i ≠ j, Z_i is not independent of Z_j given the observed network Y. This poses a huge computational challenge. The intractability of the complete log-likelihood motivates the use of variational approximation. Variational methods are well studied in literature (Blei et al., 2017). The basic idea is to posit a tractable auxiliary distribution A_γ (z)≡ P(Z = z) for the latent variables Z and find the optimal setting for variational parameters γ that minimizes the Kullback-Liebler divergence between the approximation and true distribution. We use this auxiliary distribution to construct a tractable lower bound of the log-likelihood using Jensen’s inequality and then maximize this lower bound, yielding approximate maximum likelihood estimates.

$$\begin{gathered} \ell (\mathit{\theta },\mathit{\pi },\mathit{f})=\log \left(\sum _{\mathbf{z}\in {\{1,\mathrm{...},K\}}^n}\frac{P_{\mathit{\theta },\mathit{f}}(\mathbf{Y}=\mathbf{y}|\mathbf{z})P_{\mathit{\pi }}(\mathbf{Z}=\mathbf{z})}{A_{\mathit{\gamma }}(\mathbf{z})}{A}_{\mathit{\gamma }}(\mathbf{z})\right) \\ \ge \sum _{\mathbf{z}\in {\{1,\dots ,K\}}^n}\log \left(\frac{P_{\mathit{\theta },\mathit{f}}(\mathbf{Y}=\mathbf{y}|\mathbf{z})P_{\mathit{\pi }}(\mathbf{Z}=\mathbf{z})}{A_{\mathit{\gamma }}(\mathbf{z})}\right)A_{\mathit{\gamma }}(\mathbf{z}) \\ =\mathrm{ELBO}(\mathit{\theta },\mathit{\gamma },\mathit{\pi },\mathit{f}) \end{gathered}$$ (10)

The derivation in (10) uses an auxiliary distribution and also Jensen’s inequality. We choose the variational distribution A(Z) from the mean-field family as,

$$A_{\mathit{\gamma }}(\mathbf{Z})=\prod _{i=1}^n{P}_{{\mathit{\gamma }}_i}\left({\mathbf{Z}}_i\right)=\prod _{i=1}^n\prod _{k=1}^K{\gamma }_{ik}^{z_{ik}}$$ (11)

where ∀ i ∈{1,…, n}, k ∈{1,…, K}, we have 0 ≤ γ_ik ≤ 1 with the constraint $\sum _{k=1}^K{\gamma }_{ik}=1$ and $z_{ik}=1_{z_i=k}$. This class of probability distributions A_γ considers independent laws through different nodal memberships. With the definition of A(Z) in (11), we derive the following effective lower bound (ELBO).

$$\begin{gathered} \mathrm{ELBO}(\mathit{\theta },\mathit{\gamma },\mathit{\pi },\mathit{f}) \\ =\sum _{i=1}^N\sum _{j=i+1}^N\sum _{k=1}^K\sum _{l=1}^K\{{\gamma }_{ik}{\gamma }_{jl}\left[1_{e_{ij}\ne 0}\log \left(p_{kl}\right)+1_{e_{ij}=0}\log \left(1-p_{kl}\right)\right] \\ +1_{e_{ij}\ne 0}\left[\log \left(f_{kl}\left(w_{ij}\right)\right)-\left({\int }_{\mathcal{X}}{f}_{kl}(u)du-1\right)\right]\} \\ +\sum _{i=1}^N\sum _{k=1}^K{\gamma }_{ik}\left[\log \left({\pi }_k\right)-\log \left({\gamma }_{ik}\right)\right] \end{gathered}$$ (12)

Variational E-step: we maximize ELBO (θ, γ, π, f ) in (12) to obtain γ^(t):

$${\mathit{\gamma }}^{(t)}=\underset{\gamma }{\arg \max }\mathrm{ELBO}\left({\mathit{\theta }}^{(t-1)},\mathit{\gamma },{\mathit{\pi }}^{(t-1)},{\mathit{f}}^{(t-1)}\right)$$ (13)

The direct maximization of ELBO in (13) is difficult, since the lower bound depends on the products γ_ikγ_jl and, therefore the fixed-point updates of γ_ik depend on all other γ_jl (Daudin et al., 2008). To separate the parameters in this maximization problem, we adopt an MM algorithm that involves constructing a surrogate (minorizing) function and optimizing it iteratively (Hunter and Lange, 2004). The surrogate function Q must satisfy the following properties to qualify as a valid minorizing function.

$$Q\left({\mathit{\theta }}^{(t)},{\mathit{\gamma }}^{(t)},{\mathit{\pi }}^{(t)},{\mathit{f}}^{(t)};\mathit{\gamma }\right)\le \mathrm{ELBO}\left({\mathit{\theta }}^{(t)},\mathit{\gamma },{\mathit{\pi }}^{(t)},{\mathit{f}}^{(t)}\right),\forall \mathit{\gamma }$$ (14)

$$Q\left({\mathit{\theta }}^{(t)},{\mathit{\gamma }}^{(t)},{\mathit{\pi }}^{(t)},{\mathit{f}}^{(t)};{\mathit{\gamma }}^{(t)}\right)=\mathrm{ELBO}\left({\mathit{\theta }}^{(t)},{\mathit{\gamma }}^{(t)},{\mathit{\pi }}^{(t)},{\mathit{f}}^{(t)}\right)$$ (15)

First we note that for all (θ_kl)_{1≤k≤l≤K} we have log(p_kl ) < 0 and log(1 − p_kl ) < 0 which gives rise to following inequalties using the arithmetic geometric mean inequality:

$${\gamma }_{ik}{\gamma }_{jl}\log \left(p_{kl}\right)\ge ({\gamma }_{ik}^2\frac{{\widehat{\gamma }}_{jl}}{2{\widehat{\gamma }}_{ik}}+{\gamma }_{jl}^2\frac{{\widehat{\gamma }}_{ik}}{2{\widehat{\gamma }}_{jl}})\log \left(p_{kl}\right)$$ (16)

$${\gamma }_{ik}{\gamma }_{jl}\log \left(1-p_{kl}\right)\ge \left({\gamma }_{ik}^2\frac{{\widehat{\gamma }}_{jl}}{2{\widehat{\gamma }}_{ik}}+{\gamma }_{jl}^2\frac{{\widehat{\gamma }}_{ik}}{2{\widehat{\gamma }}_{jl}}\right)\log \left(1-p_{kl}\right)$$ (17)

with equality if ${\gamma }_{ik}={\widehat{\gamma }}_{ik}$ and ${\gamma }_{jl}={\widehat{\gamma }}_{jl}$. Also the concavity of the logarithm function gives rise to following inequality (Vu et al., 2013)

$$-\log \left({\gamma }_{ik}\right)\ge -\log \left({\widehat{\gamma }}_{ik}\right)-\frac{{\gamma }_{ik}}{{\widehat{\gamma }}_{ik}}+1$$ (18)

We construct the surrogate function that satisfies (14) and (15) using the inequalities (16), (17) and (18), thus guaranteeing the ascent property of ELBO.

$$\begin{gathered} \text{Q}\left({\mathit{\theta }}^{(t)},{\mathit{\gamma }}^{(t-1)},{\mathit{\pi }}^{(t)},{\mathit{f}}^{(t)};\mathit{\gamma }\right) \\ =\sum _{i=1}^{n-1}\sum _{j=i+1}^n\sum _{k=1}^K\sum _{l=1}^K\{\left({\gamma }_{ik}^2\frac{{\gamma }_{jl}^{(t-1)}}{2{\gamma }_{ik}^{(t-1)}}+{\gamma }_{jl}^2\frac{{\gamma }_{ik}^{(t-1)}}{2{\gamma }_{jl}^{(t-1)}}\right)[1_{e_{ij}\ne 0}\log \left(p_{kl}^{(t)}\right)+ \\ {1}_{e_{ij}=0}\log \left(1-p_{kl}^{(t)}\right)]+1_{e_{ij}\ne 0}[\log \left(f_{kl}^{(t)}\left(w_{ij}\right)\right)-\left({\int }_{\chi }\left(f_{kl}^{(t)}(u)du-1\right)\right]\} \\ +\sum _{i=1}^n\sum _{k=1}^K{\gamma }_{ik}\left[\log \left({\pi }_k^{(t)}\right)-\log \left({\gamma }_{ik}^{(t-1)}\right)-\frac{{\gamma }_{ik}}{{\gamma }_{ik}^{(t-1)}}+1\right] \end{gathered}$$ (19)

To maximize (19), we solve n separate quadratic programming problems of K variables γ_i under the constraints γ_ik ≥ 0, ∀ k ∈{1,…, K} together with $\sum _{k=1}^K{\gamma }_{ik}=1$.

M-step: we first maximize the (12) with respect to π and θ. We have a closed-form update for π as ${\pi }_k^{(t)}=\sum _{i=1}^n{\gamma }_{ik}^{(t)}/n$. To update θ, we maximize (12) using the modified Newton-Raphson method with line search to guarantee the ascent property (Dennis Jr and Schnabel, 1996). Now, it remains to update block densities f. To this end, we use (6) and the approximation of log(f_kl(.)). For simplicity, we use a local polynomial approximation such that log(f_kl(.)) can be approximated by a low-degree polynomial ζ_kl in a neighborhood of the fitting point w (Loader, 1996): $\log \left(f_{kl}(u)\right)\approx {\zeta }_{kl}(u-w)=\sum _{m=0}^p{\beta }_m^{(kl)}{(u-w)}^m$. With this approximation, we rewrite (7) as

$$\begin{gathered} \ell (\mathit{\theta },\mathit{\beta },w;\mathit{Y}|\mathit{Z}) \\ =\sum _{i=1}^{n-1}\sum _{j=i+1}^n\{\left[1_{e_{ij}\ne 0}\log \left(p_{z_i{z}_j}\right)+1_{e_{ij}=0}\log \left(1-p_{z_i{z}_j}\right)\right]+1_{e_{ij}\ne 0}[K_h\left(w_{ij}-w\right) \\ {\zeta }_{z_i{z}_j}\left(w_{ij}-w\right)-\left({\int }_{\mathcal{X}}{K}_h(u-w)\exp \left({\zeta }_{z_i{z}_j}(u-w)\right)du-1\right)]\}, \end{gathered}$$

and then derive the corresponding ELBO as

$$\begin{gathered} \text{ELBO}(w;\mathit{\theta },\mathit{\beta },\mathit{\gamma }) \\ =\sum _{i=1}^{n-1}\sum _{j=i+1}^n\sum _{k=1}^K\sum _{l=1}^K\{{\gamma }_{ik}{\gamma }_{jl}(\left[1_{e_{ij}\ne 0}\log \left(p_{kl}\right)+1_{e_{ij}=0}\log \left(1-p_{kl}\right)\right]+1_{e_{ij}\ne 0} \\ \times \left[K_h\left(w_{ij}-w\right){\zeta }_{kl}\left(w_{ij}-w\right)-\left({\int }_{\mathcal{X}}{K}_h(u-w)\exp \left({\zeta }_{kl}(u-w)\right)du-1\right)\right])\} \\ +\sum _{i=1}^N\sum _{k=1}^K{\gamma }_{ik}\left[\log \left({\pi }_k\right)-\log \left({\gamma }_{ik}\right)\right] \end{gathered}$$ (20)

We maximize (20) over a sequence of grid points to estimate the block densities f_kl.

Remark 3. We note that, the estimated densities in the M-step directly affect the optimization over variational parameters γ, which can seen by the ELBO constructed after the variational approximation. Given that cluster memberships are usually estimated by the hard clustering over these parameters, weights also affect the clusters.

4. Simulation Studies

In this section, we conduct simulation studies to examine our proposed non-parametric methods. The general procedure that we adopt to simulate entails the following steps:

1. First we simulate the membership indicators for all nodes from multinomial distribution with parameter vector π corresponding to uniform mixture proportions.

2. We simulate the binary adjacency matrix by simulating dyads in the static network given the cluster membership indicators of nodes. While simulating these dyads we use the network parameters with two settings ${\theta }_{s_1}=(-1,1)$ and ${\theta }_{s_2}=(-0.5,0.5)$. The first setting corresponds to well separated clusters on the basis of density of edges while second setting considers the more extreme case when the clusters are relatively close.

3. For each node pair with an edge, we simulate the weight on that edge using true distribution with block parameter that depends on their cluster memberships.

We consider two distributions separately: Normal and Gamma. For space consideration, we include all results about Gamma distributions in the supplementary file.

We compare three different model-based clustering methods in each simulation, which are based on binary ERGM, proposed nonparametric weighted ERGM and “oracle” parametric weighted ERGM (Desmarais and Cranmer, 2012) with the correct specification of weight distributions. We consider different node sizes from 100 to 500 and 100 repetitions. Before proceeding, we introduce several average metrics to measure clustering and model parameters estimation performance for different simulation settings over 100 replications. First, to assess the clustering performance, we calculate the log of Rand Index (logRI). The measure $\mathrm{RI}(\mathit{z},\widehat{\mathit{z}})$ calculates the proportion of pairs whose estimated labels correspond to the true labels in terms of being assigned to the same or different groups (Rand, 1971). We calculate logRI as,

$$\log \text{RI}=\log (\frac{1}{\left(\begin{array}{l}n\hfill \\ 2\hfill \end{array}\right)}\sum _{i<j}\left(I\left\{z_i=z_j\right\}I\left\{{\widehat{z}}_i={\widehat{z}}_j\right\}+I\left\{z_i\ne z_j\right\}I\left\{{\widehat{z}}_i\ne {\widehat{z}}_j\right\}\right))$$

Next, to assess the performance of the estimators of network parameters θ, we consider log of square root of the average squared error (logRASE),

$$\log {\text{RASE}}_{\mathit{\theta }}=\frac{1}{2}\log \left(\frac{1}{K}\sum _{k=1}^K{\left({\mathit{\theta }}_k-{\mathit{\theta }}_k\right)}^2\right)$$

To assess the performance of the density estimation f, we consider the Kolmogorov-Smirnov (KS) statistic,

$${\text{KS}}_f=\underset{w}{\sup }|\mathit{f}(w)-\mathit{f}(w)|$$

Based on the metrics defined here, Figures 1 and 2 show clustering and θ estimation performance for different sparsity parameter settings averaged over 100 simulations of graphs for normal weight distributions. Corresponding figures for Gamma weight distributions look similar and have been moved to Appendix A. The differences in logRI and logRASE for ${\theta }_{s_1}$ and ${\theta }_{s_2}$ evidently confirms the expected fact that separating two very close clusters is difficult compared to well separated clusters. It appears that the both the distributions allow a reasonable recovery of the cluster membership indicators, when the graphs considered have more than 100 nodes. As expected, the node size improves the recovery of latent structure and estimation of network parameters θ in all cases. It can be observed that our proposed nonparametric ERGM outperforms the binary ERGM by a large difference and performs competitively with the oracle method (parametric ERGM with true weight distributions) for all simulation settings. We note that our proposed strategy is best suited for real world applications when the true distributions for block pairs are unknown.

Fig. 1.

Clustering Performance measured using logRI against different node sizes comparing the three models for different sparsity parameter settings under Normal weight distributions

Fig. 2.

θ Estimation Performance measured using logRASE against different node sizes comparing the three models for different sparsity parameter settings under Normal weight distributions

Figures 3 and 4 show the empirical distributions of network parameters θ for normal weight distributions. Corresponding figures for Gamma weight distributions have been moved to Appendix A. The proposed non-parametric model estimation again outperforms the binary ERGM uniformly for all settings. We also note that the contour plots for the proposed model seem really close to Oracle model, thus demonstrating the power of our approach.

Fig. 3.

Plots of empirical joint distributions of network parameters ${\theta }_{s_1}$ for Normal weight distributions over 100 simulations with 500 nodes, comparing the three models for different block distributions

Fig. 4.

Plots of empirical joint distributions of network parameters ${\theta }_{s_2}$ for Normal weight distributions over 100 simulations with 500 nodes, comparing the three models for different block distributions

Figure 5 shows the estimated block densities within 2.5 and 97.5 percentiles for node size of 500 for normal weight distributions. Corresponding figure for Gamma weight distributions have been moved to Appendix A. Comparing ${\theta }_{s_1}$ and ${\theta }_{s_2}$, it is evident that within cluster 1 density estimation improves substantially for ${\theta }_{s_2}$. This is because cluster 1 is more sparse for ${\theta }_{s_1}$ compared to ${\theta }_{s_2}$. Comparing the normal and gamma distributions, we observe that asymmetry of gamma distribution leads to underestimation of within cluster 1 estimated density. Cluster 1 is again most affected since it is most sparse. We point out here that for larger node sizes, these estimated densities will converge to true densities (Loader, 1996).

Fig. 5.

Estimated block densities for normal weight distributions.

Table 1 gives the summary of KS statistic for various simulation settings. We note that for sparse cluster 1, comparing ${\theta }_{s_1}$ and ${\theta }_{s_2}$, there is a huge improvement when the true distribution is Normal. However the improvement is only minor in case of Gamma due to asymmetry. The differences are much less substantial for other blocks, however Normal uniformly outperforms Gamma for all settings.

Table 1.

Summary of KS statistic (×10²) for the three block densities under different simulation settings for the proposed model, computed over 100 simulations of graphs with 500 nodes

		${\mathit{\theta }}_{{\mathit{s}}_{\mathbf{1}}}$		${\mathit{\theta }}_{{\mathit{s}}_{\mathbf{2}}}$
	Summary Statistic	Normal	Gamma	Normal	Gamma
Block (1,1)	Median	3.94	4.51	2.77	4.26
	Mean	4.03	4.70	2.84	4.36
Block (1,2)	Median	1.40	1.52	1.46	1.57
	Mean	1.41	1.56	1.49	1.61
Block (2,2)	Median	1.51	1.53	1.63	1.76
	Mean	1.53	1.60	1.66	1.76

5. Application to Water Pollution Analysis

In this section, we demonstrate the power of our methodology in an environmental application to study sulfate concentrations in river networks. The dataset consists of three main parts. The first part consists of approximately 865 sulfate samples measured as concentrations in the units of parts per million (ppm) over several creeks in the Ohio watershed in Pennsylvania. The source include the following online databases: the USGS National Water Information System, the Susquehanna River Basin Commission database, the EPA STORET Data Warehouse, and the Shale Network database (doi:10.4211/his-data-shalenetwork). The second part consists of the directed geographical river network in the form of locations of all streams and creeks in Pennsylvania. The directions in the network correspond to the actual river flow. The third part consists of 93 coal mine locations which are suspected to be potential polluters of the river streams posing an environmental risk. Both the latter parts are publically available at Pennsylvania Spatial Data Access (PASDA). We map the latitude and longitude of the sampling sites onto the geographical river network. The 865 sulfate sampling sites become the nodes. Thus the nodes in the network are defined through first part of the data after mapping to nearest streams in the second part. We define the edges according to the path of river flow; for sampling sites A and B, the edge is present when river can flow either from A to B or from B to A directly or through one of its sub-tributary or some intermediary stream. The path information used to define the edges entirely comes from the second part. The spatial weights are constructed as proportional to the strength of influence of the upstream site on the downstream site, measured by difference in concentrations between the sampling locations (Peterson et al., 2007). For example if the river is flowing from A to B, with measured sulfate concentrations C_A and C_B, then weight on the edge between A and B would be defined as w_AB = C_B − C_A. Note in this definition, the weights could be negative if C_B < C_A. In this context, we have transformed the geographically defined ‘river network’ to a weighted network defined above for the purpose of our analysis.

It is an important question to study the water pollution in river networks. Most of the existing spatial clustering methods rely on some “neighbourhood” metric that cluster the data points based on their spatial proximity. However, these approaches fail in a river network setup when the two points are very close spatially but still not connected by the river flow or vice-versa. These methods usually rely on some criteria to choose the number of clusters that is heuristic and not rigorously founded on model likelihood. There have been several parametric approaches to study spatial concentration gradients e.g. see Lawson and Denison (2002). However most methods fail in water pollution applications where the gradients tend to be asymmetric, heavy tailed and multimodal, with an unknown number of modes. We adopt skewness, kurtosis and Hartigan’s dip test of unimodality respectively to infer these properties over the whole sulfate network and individual clusters obtained using our model. The results, summarized in Table 2, clearly indicate that the gradients follow an asymmetric, leptokurtic and multimodal distribution for all block pairs. This calls for a generalized framework of clustering the weighted network while modeling the concentration gradients without making any distributional assumptions.

Table 2.

Distributional properties of sulfate concentration gradients

	Skewness	Kurtosis	Hartigan’s dip test p value
Whole network	0.050	3.885	< 2.2 × 10−16
Block (1,1)	0.762	6.817	< 2.2 × 10−16
Block (1,2)	−0.011	3.447	9.37 × 10−6
Block (2,2)	0.026	3.672	< 2.2 × 10−16

The proposed nonparametric weighted network models address all the aforementioned challenges. Now, we apply the proposed method to analyze the sulfate concentration network. In practice, we need to effectively choose the number of clusters (i.e., K). Since the likelihood is intractable (Biernacki et al., 2000), we follow (Daudin et al., 2008) to introduce a modified Integrated Classification Likelihood (ICL) criterion:

$${\text{ICL}}_K=\log P(\mathit{Y},\mathit{Z},\mathit{f})-(K-1)\log n-K\log \left(\frac{n(n-1)}{2}\right),$$ (21)

where the complete log-likelihood with estimated membership and densities becomes

$$\begin{gathered} \log P(\mathit{Y},\mathit{Z},\mathit{f})=\sum _{i<j}\sum _{k=1}^K\sum _{l=1}^K\{{\widehat{z}}_{ik}{\widehat{z}}_{jl}(\log P_{{\mathit{\theta }}_{z_i{z}_j}}\left(E_{ij}=e_{ij}|\mathbf{Z}=\mathbf{z}\right) \\ +\log {\widehat{f}}_{z_i{z}_j}\left(W_{ij}=w_{ij}|e_{ij}=1,\mathbf{Z}=\mathbf{z}\right))\}+\sum _{i=1}^n\sum _{k=1}^K{\widehat{z}}_{ik}\log {\widehat{\pi }}_k \end{gathered}$$

The ICL follows the philosophy of Bayesian model selection criterion. The second term in (21) penalizes for the K − 1 free parameters in the mixture proportions π. The third term accounts for the penalization of the network parameters (Matias and Miele, 2016) given the additive structure of the specified ERGM.

We choose the optimal number of clusters by maximizing the modified ICL criterion, which suggests K = 2. The estimated network sparsity parameters corresponding to these two clusters labelled C₁ and C₂ are −0.521 and −2.084 respectively indicating that C₁ has higher degree in terms of the edges ${\left(e_{ij}\right)}_{i,j\in C_1}$ on average compared to C₂. We plot the sampling sites belonging to the two clusters overlaying the potential polluter locations in Figure 6. It can be seen that coal mines 4, 5, 15, 20, 22, 38, 40, 42, 45, 46 73, 75 – 78, 82 and 88 lie directly either upstream or downstream of nodes in C₁ suggesting they may significantly affect the sulfate concentrations. As show in Table 3, C₁ consists of relatively more hubs with higher degree on average while C₂ consists more nodes with lower degree on average.

Fig. 6.

Clustered sulfate sampling sites with C₁ in red & C₂ in pink and coal mining sites in black (numbered 1–93)

Table 3.

Summary of nodes, degree and edge weights of the two clusters.

Summary	C1	C2
Number of nodes	147	718
Average Degree	18.72	5.21
Minimum	−385.50	−389.70
1st quantile	−2.00	−35.99
Median	8.081	10.24
Mean	40.41	23.66
3rd quantile	66.96	111.80
Maximum	436.40	441.00

In Figure 7(a), we plot densities estimated for these clusters. From this density plot, it is clear that C₁ has two modes (Mode 1 and Mode 3) and C₂ has one mode (Mode 2) on the positive sulfate concentrations. Based on these modes and estimated network parameters, we identify sub-regions of interest corresponding to 3 modes:

Fig. 7.

Non-parametrically estimated densities for ties within clusters and subnetworks for different modes

• Mode 1 consists of sub-regions with high degrees and higher differences of sulfate concentrations among adjacent nodes;

• Mode 2 consists of sparse sub-regions with low degrees and moderate differences of sulfate concentrations among adjacent nodes;

• Mode 3 consists of dense sub-regions with high degrees and low differences of sulfate concentrations among adjacent nodes.

The modes on positive weights indicate that the pollutant’s concentration increases downstream, thus pointing towards polluters making significant impact on the environment. We analyze three modes by plotting their corresponding sub-networks in Figure 7 and summarizing the descriptive statistics of edge weights in Table 4.

Table 4.

Descriptive statistics of edge weights of the three modes

Summary	Mode 1	Mode 2	Mode 3
Minimum	303.9	90.18	42.33
1st quantile	328.5	111.70	56.86
Median	334.60	126.80	65.46
Mean	334.80	128.30	65.42
3rd quantile	341.9	140.00	70.92
Maximum	365.8	186.20	90.02

Figures 7(c) and 7(d) gives sub-networks for the positive modes in C₁. We identify coal mine ids 10, 19, 64 and 73 directly lying on highly weighted edges around Mode 2 and emanating from the hubs. We define the hubs as nodes with an outlier degree. The outlier degree is defined in the usual sense of outliers, i.e. any degree which is greater than Q₃ + 1.5 × IQR, where Q₃ is the upper quartile of the degree distribution and IQR is the interquartile range. These nodes correspond to sampling sites usually located at junctions of several streams. The coal mines and hubs are marked black and green respectively. Since these coal mines belong to Mode 1, they are most likely to cause serious water pollution and must be investigated in a prioritized manner. Next we identify coal mine ids 2, 4, 5, 15, 32, 36, 38, 45, 49, 59 and 68 in Mode 3. These mines emanate from nodes that are connected densely to other nodes and so the affected region is larger, even though the pollution is relatively low. These mines in Mode 3 cause moderate impact over a large section of river network and must be monitored accordingly. Figure 7(b) shows the sub-network for the only positive mode in C₂. This corresponds to Mode 2 with relatively moderate to low local impacts.

We also compare the binary ERGM model with weighted model and observe that binary ERGM’s cluster 1 consists of 108 nodes, a strict subset of 147 nodes of weighted model’s C₁. Figures 7(c) and 7(d) shows these difference of nodes by yellow. These indicate that several edges would have been absent in C₁ had we used binary model, thus missing significant coal mines that lie in Mode 1. Clearly taking weights into account while clustering helps to differentiate mines between Mode 1 and Mode 3 and hence could uncover important potential polluters.

6. Discussion

We introduce a new nonparametric model-based approach for clustering large-scale weighted networks. The ERGM specification allows the flexibility to incorporate interesting network statistics and the nonparametric density function provides the robustness to study the network weights. We illustrate the power of our proposed method in a real application to study water pollution networks.

In general, our proposed method does not require a parametric specification of network weights and thus it is robust to the model mis-specification, and it can be extended to incorporate the nonparametric mixture functions or parametric constraints (DeSarbo et al., 2017; Lee and Xue, 2018). Like most nonparametric methods, when the sample size is limited, our proposed method may not perform well. The semiparametric extension such as Xue and Zou (2012, 2014) and Fan et al. (2016) seems a promising alternative to the proposed nonparametric method. Moreover, the proposed method could be computationally intensive when the number of nodes is huge. To make the proposed methods scalable, we shall follow the stochastic variational methods (Hoffman et al., 2013) to employ the minibatch sampling scheme.