On the Question of Effective Sample Size in Network Modeling: An Asymptotic Inquiry

Abstract

The modeling and analysis of networks and network data has seen an explosion of interest in recent years and represents an exciting direction for potential growth in statistics. Despite the already substantial amount of work done in this area to date by researchers from various disciplines, however, there remain many questions of a decidedly foundational nature — natural analogues of standard questions already posed and addressed in more classical areas of statistics — that have yet to even be posed, much less addressed. Here we raise and consider one such question in connection with network modeling. Specifically, we ask, “Given an observed network, what is the sample size?” Using simple, illustrative examples from the class of exponential random graph models, we show that the answer to this question can very much depend on basic properties of the networks expected under the model, as the number of vertices n_V in the network grows. In particular, adopting the (asymptotic) scaling of the variance of the maximum likelihood parameter estimates as a notion of effective sample size, say n_eff, we show that whether the networks are sparse or not under our model (i.e., having relatively few or many edges between vertices, respectively) is sufficient to yield an order of magnitude difference in n_eff, from O(n_V) to $O(n_V^2)$. We then explore some practical implications of this result, using both simulation and data on food-sharing from Lamalera, Indonesia.

1 Introduction

Since roughly the mid-1990s, the study of networks has increased dramatically. Researchers from across the sciences — including biology, bioinformatics, computer science, economics, engineering, mathematics, physics, sociology, and statistics — are more and more involved with the collection and statistical analysis of data associated with networks. As a result, statistical methods and models are being developed in this area at a furious pace, with contributions coming from a wide spectrum of disciplines. See, for example, the work of Jackson (2008), Kolaczyk (2009), and Newman (2010) for recent overviews from the perspective of economics, statistics, and statistical physics, respectively.

A cross-sectional network is typically represented mathematically by a graph, say, G = (V, E), where V is a set of n_V vertices (commonly written V = {1, …, n_V}) and E is a set of |E| edges (represented as vertex pairs (u, υ) ∈ E). Edges can be either directed (wherein (u, υ) is distinct from (υ, u)) or undirected. Prominent examples of networks represented in this fashion include the World Wide Web graph (with vertices representing web-pages and directed edges representing hyper-links pointing from one page to another), protein-protein interaction networks in biology (with vertices representing proteins and undirected edges representing an affinity for two proteins to bind physically), and friendship networks (with vertices representing people and edges representing friendship nominations in a social survey).

A great deal of attention in the literature has been focused on the natural problem of modeling networks—of the presence and absence of their edges in particular. There is by now a wide variety of network models that have been proposed, ranging from models of largely mathematical interest to models designed to be fit statistically to data. See, for example, the sources cited above or, for a shorter treatment, the review paper by Airoldi et al. (2009). The derivation and study of network models is a unique endeavor, due to a number of factors. First, the defining aspect of networks is their relational nature, and hence the task is effectively one of modeling complex dependencies among the vertices. Second, quite often there is no convenient space associated with the network, and so the type of distance and geometry that can be exploited in modeling other dependent phenomena, like time series and spatial processes, generally are not available when modeling networks. Finally, network problems frequently are quite large, involving hundreds if not thousands or hundreds of thousands of vertices and their edges. Since a network of n_V vertices can in principle have on the order of $O(n_V^2)$ edges, in network modeling and analysis — particularly statistical analysis of network data — the sheer magnitude of the network can be a critical factor in this area.

Suppose that we observe a network, in the form of a directed graph G = (V, E), where V is a set of n_V = |V| vertices and E is a set of ordered vertex pairs, indicating edges. We will focus on graphs with no self-loops: (u, u) ∉ E for any u ∈ V. Alternatively, we may think of G in terms of its n_V × n_V adjacency matrix Y, where Y_ij = 1, if (i, j) ∈ E, and 0, otherwise, with Y_ii ≡ 0. What is our sample size in this setting? At the opening workshop of the recent Program on Complex Networks, held in August of 2010 at the Statistical and Applied Mathematical Sciences Institute (SAMSI), in North Carolina, USA, this question in fact evoked three different responses:

1. it is the number of unique entries in Y, i.e., n_V(n_V − 1);

2. it is the number of vertices, i.e., n_V ; or

3. it is the number of networks, i.e., one.

Which answer is correct? And, why should it matter?

Despite the already vast literature on network modeling, to the best of our knowledge this question has yet to be formally posed much less answered. Closest to doing so are, perhaps, Frank and Snijders (1994) and Snijders and Borgatti (1999), who offer some discussion of this issue in the context of jackknife and bootstrap estimation of variance in network contexts. That this should be so is particularly curious given that the analogous questions have been asked and answered in other areas involving dependent data. In particular, the notion of an effective sample size has been found to be useful in various contexts involving dependent data, including survey sampling, time series analysis, spatial analysis, and even genetic case-control studies (Thibaux and Zwiers, 1984; Yang et al., 2011). Given a sample of size n in such contexts, an effective sample size — say, n_eff — typically is defined in connection with the variance of an estimator of interest. An understanding of n_eff, as a function of n, can help lend important insight into a variety of fundamental and inter-related concerns, including the precision with which inference can be done, the amount of information contributed by the data towards learning a parameter(s), and, more practically, the resources needed for data collection.

For example, in survey sampling, where nontrivial dependencies can arise through the use of complex sampling designs, n_eff generally is taken to be the sample size necessary under simple random sampling with replacement to obtain a variance equal to that resulting from the actual design used (e.g., Lavrakas (2008)). Alternatively, consider a simple AR(1) time series model, where X_t = μ + ϕX_t−1 + Z_t, for |ϕ| < 1 and Z_t independent and identically distributed normal random variables, with mean zero and variance σ². For a sample of size n, the sample mean X̄_n, the natural and unbiased estimator of μ, has a variance that behaves asymptotically in n like σ²/[n(1 − ϕ)²]. Contrasting this expression with σ²/n, corresponding to the case of independent and identically distributed X_t (i.e., equivalent to the case where ϕ ≡ 0), the value n_eff = n(1 − ϕ)² is sometimes interpreted as an effective sample size.

In these and similar contexts, it is often possible to show that, whereas nominally the relevant (asymptotic) variance scales inversely with the sample size n, under dependency a different scaling obtains, reflecting a combination of (a) the nominal sample size n, and, importantly, (b) the dependency structure in the data. Since networks are defined by relational data and, hence, consist of random variables that are inherently dependent, it seems not unreasonable to hope that we might similarly gain insight into the above question ‘What is the sample size?’ in a network setting, with the corresponding n_eff expected to be some function of the number of vertices n_V, modified by characteristics of the network structure itself.

Following similar practice in these other fields, therefore, we will interpret the scaling of the asymptotic variances of maximum likelihood estimates in a network model as an effective sample size. In this paper we provide some initial insight into the question of what is the effective sample size in network modeling, focusing on the impact of what is arguably the most fundamental of network characteristics — sparsity. A now commonly acknowledged characteristic of real-world networks is that the actual number of edges tends to scale much more like the number of vertices (i.e., O(n_V)) than the number of potential edges (i.e., $O(n_V^2)$). Here we demonstrate that two very different regimes of asymptotics, corresponding to responses 1 and 2 above, obtain for maximum likelihood estimates in the context of a simple case of the popular exponential random graph models, under non-sparse and sparse variants of the models. Response 3 suggests no meaningful asymptotics other than via independent replication. These may arise in some unexpected settings, such as with discrete-time Markov models for evolution of networks over time (Hanneke et al., 2010; Krivitsky and Handcock, 2014, for example). However, we do not explore this direction here.

We will also show that the notion of regime of asymptotics relates to the notion of consistency, as it applies to networks. Krivitsky et al. (2011) showed, informally, that their offset model was consistent, in the sense that if the network’s asymptotic regime agreed with the model, the coefficients of the non-offset terms would converge to some asymptotic value. Although the results of Shalizi and Rinaldo (2013) suggest that consistency may be meaningless for linear ERGMs with nontrivial dependence structure, our results, both theoretical and simulated, suggest that offsets that control the asymptotic regime of the network model can produce consistency-like properties.

As a technical aside, we note that the exponential random graph models we consider here are only relatively simple versions of those commonly used in practice. We choose to work with these models because (i) they are amenable to relatively standard tools in producing the theoretical results we require, while, nevertheless, (ii) they are sufficient in allowing us to highlight in a straightforward and illustrative manner our key finding – that the question of effective sample size in network settings can in fact be expected to be non-trivial and that the answer in general is likely to be subtle, depending substantially on basic model assumptions. That such insight may be obtained already for the simplest models in this class not only speaks to the fundamental nature of our results, but also appears to be fortunate, in that it would appear that theoretical analysis of the key quantity involved in our calculations becomes decidedly more delicate when even moderately more sophisticated models are considered. We provide further comments in this direction at the end of this paper.

The rest of this paper is organized as follows. Some background and definitions are provided in Section 2. Our main results are presented in Section 3, first for the case where edges arise as independent coin flips; second, for the case in which flips corresponding to edges to and from a given pair of vertices are dependent; and, third, for the case of triadic (friend-of-a-friend) effects, which we study via simulation. We then illustrate some practical implications of our results, through a simulation study in Section 4, exploring coverage of confidence intervals associated with our asymptotic arguments, and through application to food-sharing networks in Section 5, where we examine the extent to which real-world data can be found to support non-sparse versus sparse variants of our models. Finally, some additional discussion may be found in Section 6.

2 Background

There are many models for networks. See Kolaczyk (2009), Chapter 6, or the review paper by Airoldi et al. (2009). The class of exponential random graph models has a history going back roughly 30 years and is particularly popular with practitioners in social network analysis. This class of models specifies that the distribution of the adjacency matrix Y follow an exponential family form, i.e., p_θ(Y = y) ∝ exp (θ^⊤g(y)), for vectors θ of parameters and g(·) of sufficient statistics. However, despite this seemingly appealing feature, work in the last five years has shown that exponential random graph models must be handled with some care, as both their theoretical properties and computational tractability can be rather sensitive to model specification. See Robins et al. (2007), for example, and Chatterjee and Diaconis (2013), for a more theoretical treatment.

Here we concern ourselves only with certain examples of the simplest type of exponential random graph models, wherein the dyads (Y_ij, Y_ji) and (Y_kℓ, Y_ℓk) are assumed independent, for (i, j) ≠ (k, ℓ), and identically distributed. These independent dyad models arguably have the smallest amount of dependency to still be interesting as network models. A variant of the models introduced by Holland and Leinhardt (1981), they are in fact too simple to be appropriate for modeling in most situations of practical interest. However, they are ideal for our purposes, as they allow us to quickly obtain non-trivial insight into the question of effective sampling size in network modeling, using relatively standard tools and arguments.

Outside of Section 3.3, the models we consider are all variations of the form

$$p_{\alpha ,\beta }(Y=y)=\prod _{i<j}\frac{\text{exp}\{\alpha (y_{ij}+y_{ji})+\beta y_{ij}{y}_{ji}\}}{1+2e^{\alpha }+e^{2\alpha +\beta }}={\left(\frac{1}{1+2e^{\alpha }+e^{2\alpha +\beta }}\right)}^{(\begin{array}{c}\hfill n_V\hfill \\ \hfill 2\hfill \end{array})}\times \text{exp}\{\alpha s(y)+\beta m(y)\},$$ (1)

with sufficient statistics

$$s(y)\equiv \sum _{i<j}(y_{ij}+y_{ji})\text{and}m(y)\equiv \sum _{i<j}{y}_{ij}{y}_{ji},$$ (2)

a so-called Bernoulli model with reciprocity. The parameter α governs the propensity of pairs of vertices i and j to form an edge (i, j), and the parameter β governs the tendency towards reciprocity, forming an edge (j, i) that reciprocates (i, j). This model can be motivated from the independence and homogeneity assumptions given above by an argument analogous to that of Frank and Strauss (1986) using the Hammersley–Clifford Theorem (Besag, 1974), with dependence graph being D = {{(i, j), (j, i)} : (i, j) ∈ V² ∧ i < j}, the set of cliques of D being {{(i, j)} : (i, j) ∈ V² ∧ i ≠ j} ∪ {{(i, j), (j, i)} : (i, j) ∈ V² ∧ i < j}, and simplifying for homogeneity.

Of interest will be both this general model and the restricted model p_α ≡ p_{α, 0}, wherein β = 0 and there is no reciprocity, and not just dyads, but individual potential ties within dyads are independent. We will refer to this latter model simply as the Bernoulli model. Realizations of networks from this model without and with reciprocity (holding expected edge count s(y) fixed) are given in Figure 1(a) and (b), respectively.

Figure 1.

Sampled networks drawn from four configurations of (1). (a) shows a realization from a model with expected mean degree 1 on 100 vertices, and no reciprocity effect. (b) shows a realization from model with the same network size and mean degree as (a), but with reciprocity parameter β set such that the expected number of mutual ties is 25. (c) is a realization of the model from (a), scaled to 200 vertices, preserving density; while (d) preserves mean degree.

Importantly, in both the Bernoulli model and the Bernoulli model with reciprocity, we will examine the question of effective sample size under both the original model parameterization and a reparameterisation in which parameter(s) are shifted by a value log n_V. Krivitsky et al. (2011) introduced such shifts in an undirected context as a way of adjusting models like (1) for network size such that realizations with fixed α and β would produce network distributions with asymptotically constant expected mean degree, E_α,β[2s(Y)/n_V], for varying n_V. That is, a configuration (α, β) that would produce a typical n_V = 100 realization like that in Figure 1a would produce an n_V = 200 realization like that in Figure 1d. The model’s baseline asymptotic behavior is to have a constant expected density, E_α,β[2s(Y)/{n_V(n_V − 1)], such that a parameter configuration that would produce a network like 1a for n_V = 100 would produce a network like 1c for n_V = 200.

In a directed context, “degree” of a given vertex i is ambiguous, as it can refer to the number of ties that vertex makes to others (∑_j≠i Y_ij, “outdegree”), number of ties others make to that vertex (∑_j≠i Y_ji, “indegree”), the number of others to whom that vertex has at least one connection of either type (∑_j≠i max(Y_ij, Y_ji)), and the number of connections that vertex has (∑_j≠i (Y_ij + Y_ji)). In this work, we use either of the first two. Then, “mean degree” of Y is s(Y)/n_V, with mean outdegree and mean indegree trivially equal; and density is s(Y)/{n_V (n_V − 1)}.

Motivated by similar concerns, we use the presence or absence of such shifts to produce two different types of asymptotic behavior in our network model classes, corresponding to sparse (asymptotically finite mean degree) and non-sparse (asymptotically infinite mean degree) networks, respectively. Because it is widely recognized that most large real-world networks are sparse networks, this distinction is critical, and, as we show below, it has fundamental implications on effective sample size.

3 Main Results

3.1 Bernoulli Model

We first present our results for the Bernoulli model. Let p_α denote the model p_α,0, as defined above, and let $p_{\alpha }^{†}$ denote the same model, but under the mapping α ↦ α − log n_V of the density parameter. Then, it is easy to show that under p_α the mean vertex in- and out-degree tends to infinity and the network density stays at logit⁻¹(α) as n_V → ∞, while under $p_{\alpha }^{†}$, the mean degree tend to e^α while the density tends to zero. In fact, the limiting in- and out-degree distributions tend to a Poisson law with the stated mean.

From the perspective of traditional random graph theory, the offset model of Krivitsky et al. (2011) is asymptotically equivalent to the standard formulation of an Erdős-Rényi random graph, in which the probability of an edge scales like e^α/n_V. Alternatively, from the perspective of social network theory, it is use-ful to examine the log-odds that Y_ij = 1, conditional on the status of all other potential edges. Defining Y_[−ij] to be the network Y with edge (i, j) removed if present, this can be expressed as

$$\text{logit}p(Y_{ij}=1|Y_{[-ij]}=y_{[-ij]})\equiv \text{log}\frac{p(Y_{ij}=1|Y_{[-ij]}=y_{[-ij]})}{p(Y_{ij}=0|Y_{[-ij]}=y_{[-ij]})}.$$

This quantity goes from being a constant value α under p = p_α to a value α − log n_V under $p_{\alpha }^{†}$. This reflects the intuition that as long as there is a cost associated with forming and maintaining a network tie, an individual will be able to maintain ties with a shrinking fraction of the network as the network grows, with the average number of maintained ties being unaffected by the growth of the network beyond a certain point (Krivitsky et al., 2011).

Given the observation of a network Y randomly generated with respect to either of these models, initial insight into the effective sample size can be obtained by studying the asymptotic behavior of the Fisher information, which we denote ℐ(α) and ℐ^†(α) under p_α and $p_{\alpha }^{†}$, respectively. Straightforward calculation shows that while

$$ℐ(\alpha )=\left(\begin{array}{c}\hfill n_V\hfill \\ \hfill 2\hfill \end{array}\right)\frac{2e^{\alpha }}{{(1+e^{\alpha })}^2},$$

in contrast,

$${ℐ}^{†}(\alpha )=\left(\begin{array}{c}\hfill n_V\hfill \\ \hfill 2\end{array}\right)\frac{2e^{\alpha }/n_V}{{(1+e^{\alpha }/n_V)}^2}\approx n_V{e}^{\alpha }.$$

So $ℐ(\alpha )=O(n_V^2)$, while ℐ(α)^† = O(n_V), a difference by an order of magnitude.

The implications of this difference are immediately apparent when we consider the asymptotic behavior of the maximum likelihood estimates of α under the two models.

Theorem 1

Let α̂ and α̂^† denote the maximum likelihood estimates of the parameter α₀ under the models p_α0 and $p_{{\alpha }_0}^{†}$, respectively, where α₀ ∈ [α_min, α_max], for finite α̂_min, α_max. Then under the model p_α0, the estimator α̂ is ${\left(\begin{array}{c}\hfill n_V\hfill \\ \hfill 2\hfill \end{array}\right)}^{1/2}$-consistent for α₀, and

$${\left(\begin{array}{c}\hfill n_V\hfill \\ \hfill 2\hfill \end{array}\right)}^{\frac{1}{2}}(\hat{\alpha }-{\alpha }_0)\to N(0,{\left\{\frac{2e^{{\alpha }_0}}{{(1+e^{{\alpha }_0})}^2}\right\}}^{-1}),$$

while under the model $p_{{\alpha }_0}^{†}$, the estimator α̂^† is $n_V^{1/2}$-consistent for α₀, and

$$\sqrt{n_V}({\hat{\alpha }}^{†}-{\alpha }_0)\to N(0,e^{-{\alpha }_0}).$$

The proof of these results uses largely standard techniques for asymptotics of estimating equations, but with a few interesting twists. Note that, for fixed n_V, the dyads (Y_ij, Y_ji) constitute n_V (n_V − 1)/2 independent and identically distributed bivariate random variables under both p_α and $p_{\alpha }^{†}$. Consistency of the estimators in both cases can be argued by verifying, for example, the conditions of Theorem 5.9 of van der Vaart (2000) for consistency of estimating equations. Similarly, the proof of asymptotic normality of the estimators can be based on the usual technique of a Taylor series expansion of the log-likelihood and, due to the fact that we have assumed an exponential family distribution, the asymptotic normality of the sufficient statistic s(y) in (2). However, in the case of the sparse model $p_{\alpha }^{†}$, the dyads {(Y_ij, Y_ji)}_i<j follow a different distribution for each n_V, and therefore an array-based central limit theorem is required to show the asymptotic normality of s(y). But since increasing the number of vertices from, say, n_V − 1 to n_V, as n_V → ∞, increases the number of dyads in our model by n_V − 1, a standard triangular array central limit theorem is not appropriate here. Rather, a double array central limit theorem is needed, such as Theorem 7.1.2 of Chung (2001). A full derivation is provided in the supplementary materials.

3.2 Bernoulli Model with Reciprocity

From Theorem 1 we see that the effective sample size n_eff in this context can be either on the order of n_V or of $n_V^2$, depending on the scaling of the assumed model, i.e., on whether the model is sparse or not. From a non-network perspective, these results can be largely anticipated by the rescaling involved, in that the transformation α ↦ α − log n_V induces a rescaling of the expected number of edges by $n_V^{-1}$. Now, however, consider the full Bernoulli model with reciprocity, p_{α, β}, defined in (1). Even with just two parameters the situation becomes notably more subtle.

Let ℐ(α, β) be the 2 × 2 Fisher information matrix under this model. Then calculations (not shown) completely analogous to those required for our previous results show that $ℐ(\alpha ,\beta )=O(n_V^2)$ and, similarly, asymptotic properties of the maximum likelihood estimate of (α, β) analogous to those for p_α hold.

Let us focus then on sparse versions of p_α,β. The offset used previously, i.e., mapping α to α − log n_V, is not by itself satisfactory. Call the resulting model $p_{\alpha ,\beta }^{†}$. Standard arguments show that the limiting in- and out-degree distributions under this model will be Poisson with mean parameter e^α. On the other hand, the expected number of reciprocated out-ties a vertex has, ${\mathrm{E}}_{\alpha ,\beta }^{†}[2m(Y)/n_V]$, behaves like e^2α+β/n_V, and therefore tends to zero as n_V → ∞. Thus, β plays no role in the limiting behavior of the model, and, indeed, reciprocity vanishes. This fact can also be understood through examination of the Fisher information matrix, say ℐ^†(α, β), in that direct calculation shows

$${ℐ}^{†}(\alpha ,\beta )=\left[\begin{array}{cc}\hfill O(n_V)\hfill & \hfill O(1)\hfill \\ \hfill O(1)\hfill & \hfill O(1)\hfill \end{array}\right].$$

That is, only the information on α grows with the network. Under $p_{\alpha ,\beta }^{†}$, only the affinity parameter α can be inferred in a reliable manner.

However, the same intuition that suggests that as the network becomes larger, a given vertex i will have an opportunity for contact with a smaller and smaller fraction of it also suggests that if there is a preexisting relationship in the form of a tie from j to i, such an opportunity likely exists regardless of how large the network may be. This, as well as direct examination of the exact expression for the information matrix ℐ^†(α, β), suggests that the − log n_V penalty on tie logprobability should not apply to reciprocating ties, which may be implemented by mapping β ↦ β + log n_V. Call this model, in which $p_{\alpha ,\beta }^{†}$ is augmented with this additional offset for β, the model $p_{\alpha ,\beta }^{‡}$. The corresponding conditional log-odds of an edge now have the form

$$\text{logit}{p}_{\alpha ,\beta }^{‡}(Y_{ij}=1|Y_{[-ij]}=y_{[-ij]})=\{\begin{array}{cc}\alpha -\text{log}{n}_V,\hfill & \text{if}{y}_{ji}=0,\hfill \\ \alpha +\beta ,\hfill & \text{if}{y}_{ji}=1,\hfill \end{array}$$

which exactly captures the intuition described.

It can be shown that under $p_{\alpha ,\beta }^{‡}$ we have ℐ^‡(α, β) = O(n_V), indicating that information on both parameters grows at the same rate in n_V. It can also be shown that the limiting in-and out-degree distribution is now Poisson with mean parameter e^α + e^2α+β, and that ${\mathrm{E}}_{\alpha ,\beta }^{‡}[2m(Y)/n_V]$ tends to e^2α+β. So, both parameters play a role in the limiting behavior of the model and the additional offset induces an asymptotically constant expected per-vertex reciprocity in addition to asymptotically constant expected mean degree.

Finally, we have the following analogue of Theorem 1.

Theorem 2

Let (α̂^‡, β̂^‡) denote the maximum likelihood estimate of the parameter (α₀, β₀) under the model $p_{{\alpha }_0,{\beta }_0}^{‡}$, where (α₀, β₀) ∈ [α_min, α_max] × [β_min, β_max], for finite α_min, α_max, β_min, β_max. Then (α̂^‡, β̂^‡) is $n_V^{1/2}$-consistent for (α₀, β₀), and

$$\sqrt{n_V}\left(\begin{array}{c}{\hat{\alpha }}^{‡}-{\alpha }_0\hfill \\ {\hat{\beta }}^{‡}-{\beta }_0\hfill \end{array}\right)\to N(0,e^{-{\alpha }_0}[\begin{array}{cc}\hfill 1\hfill & \hfill -2\hfill \\ \hfill -2\hfill & \hfill 4+2e^{-{\alpha }_0-{\beta }_0}\hfill \end{array}\left]\right).$$

Proof of this theorem, using arguments directly analogous to those of Theorem 1, may be found in the supplementary materials. From the theorem we see that under the sparse model $p_{\alpha ,\beta }^{‡}$, as under $p_{\alpha }^{†}$, the effective sample size n_eff is n_V.

3.3 Triadic effects

Although there has been some work on obtaining closed-form asymptotics for ERGMs with triadic — friend-of-a-friend — effects (Chatterjee et al., 2011) or showing that they might not exist (Shalizi and Rinaldo, 2013), these results do not appear to be directly applicable to the per-capita asymptotic regimes that we consider in this work. In this section, we thus use simulation in an attempt to extend the intuition developed in Section 3.2 — that reciprocating ties should not be “penalized” for the network size — to these triadic effects. For the sake of simplicity, we will consider undirected networks only.

A tie between i and k and a tie between k and j — i.e., that i knows k and k knows j — should create a preexisting relationship between i and j. That is, k can “introduce” i and j regardless of how large the network is otherwise. Thus, given i − k − j relationships, a potential relationship between i and j should not be penalized for network size (though i − k − j themselves are); and more such two-paths (i.e., i − k′ − j) should have no further effect on this penalty. This suggests an offset on the statistic called the transitive ties (Snijders et al., 2010, eq. 8), or, equivalently, Geometrically-Weighted Edgewise Shared Partners (GWESP) (Morris et al., 2008) with parameter α fixed at 0, i.e.,

$$t(y)=\sum _{i<j}{y}_{ij}\underset{k}{\text{max}}(y_{ik},y_{jk}).$$ (3)

Unlike the more familiar count of the number of triangles (∑_i<j<k y_ijy_iky_jk), t(y) only considers whether a two-path exists, not how many of them there are. (This also makes it far less prone to ERGM degeneracy. (Schweinberger, 2011))

Consider the following model, with edge count and transitive tie count (3):

$$P_{\alpha ,\gamma }(Y=y)\propto \text{exp}\{-\text{log}(n_V)({\alpha }^{*}s(y)-{\gamma }^{*}t(y))+{\alpha }_{0}s(y)+{\gamma }_{0}t(y)\}.$$ (4)

As with $p_{\alpha }^{†}$, the coefficient on s is penalized by network size, in the form of log(n_V)α^⋆, with α^⋆ being 1 in $p_{\alpha }^{†}$. However, the penalty is then partially negated by increasing the coefficient on t by log(n_V)γ^⋆. This means that, on a sparse network,

$$\text{logit}{p}_{\alpha ,\gamma }(Y_{ij}=1|Y_{[-ij]}=y_{[-ij]})\approx \{\begin{array}{cc}{\alpha }_0-{\alpha }^{*}\text{log}{n}_V,\hfill & \text{if}\neg {\exists }_{k\ne i,j}{y}_{ik}{y}_{kj}=1,\hfill \\ {\alpha }_0+3{\gamma }_0-({\alpha }^{*}-3{\gamma }^{*})\text{log}{n}_V,\hfill & \text{if}{\exists }_{k\ne i,j}{y}_{ik}{y}_{kj}=1.\hfill \end{array}$$

The reason that this is an approximation is that, on an otherwise empty network having ties (i, k) and (k, j), adding a tie (i, j) creates not one but three transitive ties, by making all three of the ties transitive (hence the coefficient 3 on the γs). However, as the network becomes more dense, (i, k) and/or (k, j) may already be transitive when (i, j) is added, so only two or one transitive ties might be created.

Therefore, on a sufficiently sparse network (i.e., sufficiently large n_V for a given mean degree), in order to cancel the network size penalty for a tie (i, j) but retain it for (i, k) and (k, j), γ^⋆ = α^⋆/3. With α^⋆ = 1 per the same reasoning as before, this means that our heuristic suggests that γ^⋆ ≈ 1/3. We verify this empirically as follows. Define t′(y)— per-capita transitive ties — as

$$t\prime (y)=\frac{1}{n_V}\sum _i\sum _{j\ne i}{y}_{ij}\underset{k\ne i,j}{\text{max}}(y_{ik}{y}_{jk}).$$

In other words, for each vertex i, the number of its neighbors who have ties to at least one other neighbor of i is counted, and the resulting measures averaged over all actors in the network. It can be shown easily that t′(y) ≡ 2t(y)/n_V.

We can then ask if there exist constant values of α^⋆ and γ^⋆ that produce stable mean degree (2s(y)/n_V) and stable per-capita transitive ties (t′(y)). We constructed a series of 40 networks, sized from 100 to 12,000, logarithmically spaced, for each of three configurations of mean degree, 6, 9, and 12; and two levels of per-capita transitivity for each: 1/2 of the mean degree and 1/4 of the mean degree. (This was done because per-capita transitivity cannot exceed the mean degree.) For each combination of n_V, 2s/n_V, and t′/(2s/n_V), we used simulated annealing to construct a network y with these statistics, and then fit an ERGM p_α,γ (without offsets) to it to obtain point estimates for what is effectively (−log(n_V)α^⋆ + α₀, log(n_V)γ^⋆ + γ₀). The calculations were performed using the ergm package (Handcock et al., 2014; Hunter et al., 2008) for the R computing environment (R Core Team, 2013).

We show the results in Figure 2. Our intuition seems to be confirmed, to the extent that our network sizes are sufficiently large to confirm or disconfirm it. The trend in both parameter estimates appears to be more linear (in log n_V) as n_V increases, suggesting that unique α^⋆ and γ^⋆ exist. For α̂, the asymptotic slope (i.e., −α^⋆) is very close to −1 regardless of the mean degree and the amount of transitivity, and for α̂, the slope (i.e., γ^⋆) decreases as log n_V increases, though it does not quite obtain the exact value of 1/3 for the network sizes considered. (E.g., considering only n_V > 5,000, 2s(y)/n_V = 6, and t′(y)/(2s(y)/n_V) = 1/4 — the fastest-converging configuration — gave the slope of 0.35.)

Figure 2.

Maximum likelihood estimates from fitting p_α,γ to networks with a variety of sizes, densities (distinguished by color), and levels of transitivity (distinguished by plotting symbol). Note that the horizontal axis is plotted on the logarithmic scale.

Notably, even though given a particular value of the sufficient statistic (s(y), t(y)), the natural parameters (α, γ) would be determined exactly, we have to use Monte Carlo MLE (Hunter and Handcock, 2006) to estimate them, so there is some noise in the point estimates.

Overall, it appears that the coefficients of sparser networks with weaker transitivity tend to approach linearity faster. Thus, we performed a follow-up simulation study, this one with mean degree 2, transitivity proportion 1/8, and 40 values of n_V between 10,000 and 40,000, logarithmically spaced.

Based on all of the values of n_V considered, α̂^⋆ = 1.00037 (95% CI: (1.00029, 1.00044)) and γ̂^⋆ = 0.3377 (95% CI: (0.3369, 0.3386)), closer to the theoretical values of 1 and 1/3 than the smaller network sizes. The confidence intervals do not include the theoretical values, but we would not expect the asymptotic values to be attained for any finite network size. Indeed, there is evidence of nonlinearity in that range (P-value of predictor log(n_V)² term is < 0.0001 for the α^⋆ response and 0.04 for the γ^⋆ response, with negative coefficient for both). Furthermore, fitting only the 20 data points with n_V > 20,000 produces (α̂^⋆, γ̂^⋆) = (1.000072, 0.3347), and fitting only the 10 data points with n_V > 29,000, (α̂^⋆, γ̂^⋆) = (1.00030, 0.3334).

This very strongly suggests meaningful and interpretable asymptotic behavior for triadic closure ERGM terms as well. In particular, the asymptotic linearity with a known coefficient suggests a form of consistency for “intercepts” α₀ and γ₀, as it is they that control the asymptotic mean degree and per-vertex amount of triadic closure in (4).

To relate this to the notion of effective sample size n_eff used earlier, defined through the scaling of the information matrix ℐ(α, γ), we simulated the sufficient statistics from the above-described fits. For an exponential family, the variance covariance matrix of sufficient statistics under the MLE approximates the information matrix (Hunter and Handcock, 2006, eq. 3.5, for example). We find that the entries of ℐ̂(α̂, γ̂)/n_V = Var_{α̂, γ̂} ([s(Y), t(Y)]) /n_V do not exhibit any trend at all as a function of n_V, for fixed mean degree and per-vertex transitivity. (In particular, for a linear trend, P-values are 0.31, 0.49, and 0.41 for Var(s(Y))/n_V, Var(t(Y))/n_V, and Cov(s(Y), t(Y))/n_V, respectively. Exploratory plots do not show any pattern, except for greater variability in estimates of variance for higher n_V.) This strongly suggests that the asymptotics of the model (4) have an effective sample size n_eff of n_V as well.

4 Coverage of Wald Confidence Intervals

Our asymptotic arguments in Section 3 were developed primarily for the purpose of establishing the scaling associated with the asymptotic variance, so as to provide insight into the question of effective sample size — our main focus here. However, the asymptotically normal distributions we have derived are of no little independent interest themselves, as they serve as as a foundation for doing formal inference on the model parameters in practice. By way of illustration, here we explore their use for constructing confidence intervals, particularly those based on Theorem 2: under a model $p_{\alpha ,\beta }^{‡}$, the Wald confidence intervals using plug-in estimators for the standard errors are ${\hat{\alpha }}^{‡}\pm z_{(1-\mathrm{C}\mathrm{L})/2}^{*}\sqrt{e^{-{\hat{\alpha }}^{‡}}/n_V}$ for α and ${\hat{\beta }}^{‡}\pm z_{(1-\mathrm{C}\mathrm{L})/2}^{*}\sqrt{e^{-{\hat{\alpha }}^{‡}}(4+2e^{-{\hat{\alpha }}^{‡}-{\hat{\beta }}^{‡}})/n_V}$.

Because our asymptotics are in N_υ = |V|, we examine a variety of network sizes. The desired asymptotic properties of the network are expressed in terms of the per-capita mean value parameters— ${\mathrm{E}}_{\alpha ,\beta }^{‡}[s(Y)/n_V]$ and ${\mathrm{E}}_{\alpha ,\beta }^{‡}[2m(Y)/n_V]$. We study two configurations:

1. ${\mathrm{E}}_{\alpha ,\beta }^{‡}[s(Y)/n_V],{\mathrm{E}}_{\alpha ,\beta }^{‡}[m(Y)/n_V])=(1,0.25)$ and

2. ${\mathrm{E}}_{\alpha ,\beta }^{‡}[s(Y)/n_V],{\mathrm{E}}_{\alpha ,\beta }^{‡}[m(Y)/n_V])=(1,0.40)$.

In other words, the expected mean outdegree is set to 1, and expected numbers of out-ties that are reciprocated are 0.25 × 2 = 0.5 and 0.40 × 2 = 0.8 per vertex, respectively. These represent two levels of mutuality, though note that even (1) reprsents substantial mutuality, especially for larger networks.

For each n_V = 10, 15, 20, …, 200, we estimate the natural parameters of the model $p_{\alpha ,\beta }^{‡}$ corresponding to the desired mean value parameters, and then simulate 100,000 networks from each configuration, evaluating the MLE and constructing a Wald confidence interval at each of level of the customary 80%, 90%, 95%, and 99%, for α and for β (individually), checking the coverage.

For some of the smaller sample sizes, the simulated network statistics for some realizations were not in the interior of their convex hull. (Barndorff-Nielsen, 1978, Thm. 9.13, p. 151) That is, their values were the maximal or minimal possible: s(y) = 0 (empty graph), s(y) = n_V(n_V − 1) (complete graph), m(y) = 0 (no edges reciprocated), and/or m(y) = s(y)/2 (every extant edge reciprocated). For those, the MLE did not exist. (For (1), the fraction was 8.2% for n_V = 10 and none of the 100,000 realizations had no MLE for n_V ≥ 55. For (2), it was 14.2% for n_V = 10 and none of the realizations had no MLE for n_V ≥ 65.)

Our results are conditional on the MLE existing. From the frequentist perspective, one might argue that if the MLE did not exist for a real dataset, we would not have reported that type of confidence interval, so it should be excluded from the simulation as well.

We report coverages for selected network sizes in Table 1 and provide a visualization in Figure 3. Overall, the 80% coverage appears to be varied — and not very conservative—while higher levels of confidence appear to be more consistently conservative, particularly for estimates of β. Coverage for α appears to oscillate as a function of network size. This is particularly noticeable for the lower confidence levels and stronger mutuality (2). Tendency of a confidence interval for a binomial proportion to oscillate around the nominal level is a known phenomenon (Brown et al., 2001, 2002, and others), though it is interesting to note that it appears to be more prominent for the density, rather than mutuality, parameter and that it appears to be stronger for stronger mutuality.

Table 1.

Simulated Theorem 2 confidence interval coverage levels for selected network sizes and two levels of reciprocity: lower (1) and higher (2).

		Coverage
		80.0%		90.0%		95.0%		99.0%
	nV	α	β	α	β	α	β	α	β
(1)	10	72.4%	77.3%	85.3%	89.8%	93.2%	95.2%	96.4%	99.4%
	20	74.5%	77.3%	86.0%	89.4%	92.9%	94.9%	98.3%	99.5%
	50	80.9%	78.8%	87.6%	89.4%	94.7%	94.8%	98.9%	99.2%
	100	77.4%	79.6%	90.0%	90.0%	94.6%	94.9%	98.9%	99.1%
	200	79.0%	79.5%	90.1%	89.8%	94.9%	94.9%	98.9%	99.0%
(2)	10	84.0%	84.2%	86.6%	89.8%	93.6%	94.3%	96.3%	98.2%
	20	81.8%	80.3%	92.8%	92.1%	95.1%	96.0%	98.1%	98.8%
	50	75.3%	79.5%	91.7%	89.4%	95.6%	95.1%	98.8%	99.0%
	100	78.5%	79.7%	91.0%	90.2%	94.5%	94.9%	99.0%	99.1%
	200	82.2%	79.9%	90.5%	89.9%	95.3%	95.1%	99.2%	99.1%

Figure 3.

Scatterplot of differences between simulated coverage and nominal coverage for the two configuration studied, as a function of network size n_V. Color denotes the nominal coverage levels, and smoothing lines have been added. Note that the differences are differences in percentage points (simulated % − nominal %), not percent differences $(\frac{\text{simulated}\%-\text{nominal}\%}{\text{nominal}\%}\times 100\%)$.

5 Example: Food-Sharing Networks in Lamalera

While the results of Section 3 are important in establishing how closely the question of effective sample size in network modeling is tied to the structural property of (non)sparseness expected of the networks modeled, there remains the important practical question of establishing in applications just which model (i.e., sparse or non-sparse) is most appropriate. While a full and detailed study of this question is beyond the scope of this work, we present here an initial exploration.

Note that, in exploring this question, we face a problem similar to that pointed out by Krivitsky et al. (2011): it requires a collection of closed networks of a variety of sizes yet substantively similar social structure. Furthermore, our results are limited to modeling density and reciprocity, so the networks should be well-approximated by this model. Here, we use data collected by Nolin (2010), in which each of 317 households in Lamalera, Indonesia was asked to list the households to whom they have given and households from whom they have received food in the preceding season. Lamalera is split, administratively, into two villages, which are further subdivided into wards, and then into neighborhoods. Nolin (2010) fit several ERGMs to the network, finding that distance between households had a significant effect on the propensity to share, as did kinship between members of the households involved. Nolin also found a significant positive mutuality effect.

In our study, we make use of the geographic effect by constructing a series of 24 overlapping subnetworks, consisting of Lamalera itself, its 2 constituent villages, 6 wards, and 15 neighborhoods, with network sizes ranging from 12 to 317. We then fit the baseline model p_α,β to each network. If p_α,β is the most realistic asymptotic regime for these data, we would expect estimates α̂ and β̂ to have no relationship to log n_V for the corresponding network. If $p_{\alpha ,\beta }^{†}$ is the most realistic, we would expect no relationship between log n_V and β̂, but an approximately linear relationship with α̂, with slope around − 1. Lastly, if $p_{\alpha ,\beta }^{‡}$ is the most realistic, we would expect the slope of the relationship between log n_V and α̂ to be around − 1 and between log n_V and β̂ to be around +1.

The estimated coefficients and the slopes are given in Figure 4. The results are suggestive. The relationship between β̂ and log n_V is clearly negative, while the relationship between β̂ and log n_V is clearly positive, and the magnitudes of both slopes are closer to 1 than to 0 (although both are far from equaling 1). Overlap between the subnetworks induces dependence among the coefficients, so it is not possible to formally test or estimate how significant this difference is. Nevertheless, the preponderance of evidence is that $p_{\alpha ,\beta }^{‡}$ is the best of the three considered. That is, a sparse model that does not enforce sparsity on reciprocating ties appears to be preferable here.

Figure 4.

Maximum likelihood estimates from fitting p_α,β to each subdivision of the Lamalera food-sharing network. Note that the horizontal axis is plotted on the logarithmic scale. Colors indicate subdivision type. The least-squares coefficients from regressing α̂ and β̂ on log n_V are −0.72 and +0.60, respectively.

A possible explanation for why the magnitudes of the slopes are substantially less than 1 is that both the argument of Krivitsky et al. (2011) and our argument in Section 3.2 rely on the assumption that the network is closed — no relationships of interest are to or from actors outside of the network observed — or, at least, that the stable mean degree and per-capita reciprocity are for the ties within it. However, while there is likely to be very little food sharing out of or into Lamalera, and relatively little between the two villages it comprises (7% of all food-sharing ties in the network are between villages), there is more sharing between the wards (28% are between wards), and even more between neighborhoods (44%). Thus, the closed-network assumption is violated. (The respective between-subdivision percentages for reciprocated ties are 6%, 22%, and 39%.) When each of the subdivisions of the network is considered in isolation, these ties are lost, so the smaller subdivisions appear, to the model, to have smaller mean degree and per-capita mutuality. (See Figure 5.) This, in turn, means that smaller subdivisions have a decreased α̂ (increasing the slope for it in Fig. 4) and, because mutual ties suffer less of this “attrition” than ties do overall, the β, after adjusting for the decreased α̂, is increased for smaller networks, thus reducing the slope for β̂ in Fig. 4. It is not unlikely that this pattern will hold in any network with an unobserved spatial structure, whose subnetworks of interest are contiguous regions in this space.

Figure 5.

Per-capita network statistics as a function of n_V. Colors indicate subdivision type. Note that the larger subdivisions have more within-subdivision ties.

6 Discussion

Unlike conventional data, network data typically do not have an unambiguous notion of sample size. The theoretical developments and the examples we have presented show that the effective sample size n_eff associated with a network depends strongly on the model assumed for how the network scales. In particular, in the case of reciprocity, whether or not the model for scaling takes into account the notion of preexisting relationship affects whether reciprocity is even meaningful for large networks. In the case of triadic effects, a similar notion — along with the inutition that as the network size changes each individual’s view of triadic closure should not — implies a specific scaling regime which, in turn, implies a specific notion of the effective sample size.

The models we study here are relatively simple examples of network models. However, with reciprocity, our work includes an important aspect that already allows us a glimpse beyond the treatments of, say, Chatterjee et al. (2011) and Rinaldo et al. (2011), for so-called beta models, where the dependency induced here by reciprocity is absent. In addition, the results for reciprocity suggest that the effective modeling of triadic (e.g., friend of a friend of a friend) effects in a manner indexed to network size requires a more complex treatment yet. However, our simulation shows, perhaps somewhat surprisingly, that if triadic closure is considered on a per-capita basis, effective sample size ultimately behaves similar to the way it does in the simpler cases.

We note that asymptotic theory supporting methods for the construction of confidence intervals for network parameters is only beginning to emerge. The most traction appears to have been gained in the context of stochastic block models (e.g., Bickel and Chen (2009), Choi et al. (2012), Celisse et al. (2012), and Rohe et al. (2011)), although progress is beginning to be had with exponential random graph models as well (e.g., Chatterjee et al. (2011), Chatterjee and Diaconis (2013), and Rinaldo et al. (2011)). Most of these works present consistency results for maximum likelihood and related estimators, with the exception of Bickel and Chen (2009), which also includes results on asymptotic normality of estimators. Our work contributes to this important but nascent area with both our theoretical developments and our simulation studies. In particular, the asymptotic regime of p_α,γ; is one that appears to neither become degenerate nor approaches Erdős-Rényi.

The lack of an established understanding of the distributional properties of parameter estimates in commonly used network models is particularly unfortunate given that a number of software packages now allow for the easy computation of such estimates. For example, packages for computing estimates of parameters in fairly general formulations of exponential random graph models routinely report both estimates and, ostensibly, standard errors, where the latter are based on standard arguments for exponential families. Unfortunately, practitioners do not always seem to be aware that the use of these standard errors for constructing normal-theory confidence intervals and tests is lacking fully formal justification. From that perspective, our work appears to be one of the first to begin laying the necessary theoretical foundation to justify practical confidence interval procedures in exponential random graph models. See Haberman (1981) for another contribution in this direction, proposed as part of the discussion of the original paper of (Holland and Leinhardt, 1981).

In order to successfully build upon our work, and extend our results to more sophisticated instances of exponential random graph models, certain technical challenges must be overcome. First, we note that our notion of effective sample size is tied directly to the asymptotic behavior of the Fisher information matrix of our model (denoted ℐ(θ) in the supplementary materials). Given that exponential random graph models are, by definition, of exponential family form, this information matrix is in principle given by the matrix of partial second derivatives of the cumulant generating function (denoted ψ in the supplementary materials, so that ℐ = ∂²ψ(θ)/∂θ∂θ^T). Due to the use of independent dyads in our theoretical work (i.e., our models are variations on Bernoulli models), the corresponding likelihoods factor over dyads, and hence the information matrices are simply proportional to powers of N_υ (i.e., linear or quadratic). This is in analogy to the canonical setting of independent and identically distributed observations. However, in more general settings beyond the case of independent dyads — including even the models with triadic effects we studied in simulation, the likelihood cannot be expected to factor in such a simple manner. Hence, the analysis of the Fisher information promises to be decidedly more subtle. In fact, there appears to be almost no work to date studying this matrix in any detail. To the best of our knowledge the only such work is the recent manuscript by Pu et al. (2013), introducing a deterministic approach to approximating this matrix (stochastic approximations may, of course, be produced using MCMC) based on a lower-bound of the cumulant generating function. This bound, however, has only an implicit representation.

Second, in the case of more general exponential random graph models than those studied here, there will be a need for a correspondingly more sophisticated central limit theorem, in order to produce results on asymptotic normality analogous to those we present for the simpler models we study. Even for our models, the tool we used was somewhat nonstandard, in that we required a double-array central limit theorem. The more general case will require a central limit theorem capable of handling the nontrivial global dependencies induced by effects even as seemingly simple as triadic closure or the like. Progress on the first point above is a likely prerequisite to understanding the nature of these dependencies sufficiently well to know just what sort of central limit theorem is required.

Finally, there is, as always with exponential random graph models, the issue of instability and degeneracy that must be kept in mind (e.g., Handcock (2003) and Chatterjee et al. (2011)). It has been discovered only relatively recently that substantial care must be taken in specifying network effects in exponential random graph models. Without such care, it is possible to produce models for which the corresponding distributions turn out to be near-degenerate and, in turn, the estimation of parameters, highly unstable. Schweinberger (2011) has recently shed important light on this issue, showing that instability and degeneracy are related to the scaling of the linear term in exponential family distributions generally and, more specifically, in exponential random graph models. These scaling results can be expected to have implications on the role that scaling necessarily plays in the types of calculations we have presented here.