Motif-guided Heterogeneous Graph Deep Generation

Abstract

The complex systems in the real-world are commonly associated with multiple types of objects and relations, and heterogeneous graphs are ubiquitous data structures that can inherently represent multi-modal interactions between objects. Generating high-quality heterogeneous graphs allows us to understand the implicit distribution of heterogeneous graphs and provides benchmarks for downstream heterogeneous representation learning tasks. Existing works are limited to either merely generating the graph topology with neglecting local semantic information or only generating the graph without preserving the higher-order structural information and the global heterogeneous distribution in generated graphs. To this end, we formulate a general, end-to-end framework - HGEN for generating novel heterogeneous graphs with a newly proposed heterogeneous walk generator. On top of HGEN, we further develop a network motif generator to better characterize the higher-order structural distribution. A novel heterogeneous graph assembler is further developed to adaptively assemble novel heterogeneous graphs from the generated heterogeneous walks and motifs in a stratified manner. The extended model is proven to preserve the local semantic and heterogeneous global distribution of observed graphs with the theoretical guarantee. Lastly, comprehensive experiments on both synthetic and real-world practical datasets demonstrate the power and efficiency of the proposed method.

1. Introduction

Graphs have emerged as an important data genre found in a wide class of applications. Researchers have devoted themselves to studying various types of graph problems, resulting in a rich literature of related papers and methods [1–7] in recent years, which can be primarily categorized into two directions: 1) graph representation learning aims at encoding graph topological and semantic information into vector space [8]; and 2) graph generation, which reversely aims at constructing graph-structured data from low-dimensional space containing the graph generation rules or distribution [9]. Many efforts have been devoted to studying both representation learning and graph generation on homogeneous graphs. However, as the superclass of the homogeneous graph, heterogeneous graphs come with different types of information attached to nodes and edges, which can contain considerably richer semantic information than homogeneous graphs [10]. Figure 1(b) shows a citation network with author, paper, venue, and term as nodes and “authorship”, “containment” and “publishment” as edges. The local semantic information based on certain combinations of node types and edge types reflect the key patterns of heterogeneous graphs [11, 12], and such combinations of nodes and edges are typically referred to as meta-path. Meta-paths characterize the rich and diverse relations among nodes [12, 13]. For example, as shown in Figure 1(b), two authors can be connected via a meta-path since they both contribute to a paper, while two authors can alternatively be connected because their papers are accepted at the same venue.

Fig. 1:

Examples of heterogeneous graphs in the academic field.

As a more powerful, realistic, and generic superclass of traditional homogeneous graphs, heterogeneous graphs have recently been intensively studied. Existing literature focuses generally on learning network representations and latent embeddings for various network mining and analytical tasks, such as meta-relation detection [14, 15], heterogeneous node embedding learning [16, 17], and heterogeneous link prediction [18]. However, the other perspective of heterogeneous graph study - heterogeneous graph generation- remains paucity. Other than providing benchmarks for many heterogeneous graph studies, realistic heterogeneous graph generation has at least two advantages: 1) generating high-quality heterogeneous graphs requires us to comprehensively capture the latent graph distribution, which can significantly enrich our understanding of the implicit properties of heterogeneous graphs; 2) generating heterogeneous graphs is helpful in specific downstream applications (e.g., recommendation system [19], knowledge graph reasoning [18], and node proximity search [11]). Given the importance of the research problem, there is only one work [20] that has tried to generate random heterogeneous graphs with handcrafted rules, which fails to decode the real data distribution underlying the observed graphs.

In the past few years, we have witnessed plenty of deep homogeneous graph generative models [2, 9, 21–23] that can learn the observed graph distribution without prescribed rules, which have shown advantages in preserving various static graph properties in the generated graphs. However, existing deep generative models designed for homogeneous graphs cannot be trivially adapted to heterogeneous graphs due to the following technical difficulties: 1) Difficulties in preserving heterogeneous semantic information. Current works for homogeneous graphs have been either using random walks as a tool to learn the graph topological distribution as learning the distribution of random walks ([22, 24]) or directly modeling an overall distribution of the edges ([23, 25]) over the homogeneous graphs. However, objects in heterogeneous graphs are interconnected via various meta-paths, as shown in Figure 1(c). As meta-paths carry the complex local semantic information, adapting current works to the heterogeneous graph scenario without any elaborations on meta-path would bring difficulties in learning and preserving the distribution of such complex semantic patterns spanning different graph entities (i.e., edges and nodes) in the newly generated heterogeneous graphs. 2) Difficulties in preserving heterogeneous higher-order structural information. In the study of heterogeneous graphs, meta-paths may also fall short of expressing more intricate relationships among nodes in heterogeneous graphs. As marked in Figure 1(b), some common and symmetric higher-order structures spanning meta-paths will likely be observed repeatedly, which forms a triangle or orbit structure (e.g., one author writes two papers that are accepted by the same venue, and two papers of an author focus on the same research topic). These higher-order connectivity patterns are known to be important in understanding the structure and organization of heterogeneous networks, and many works [26, 27] have proposed to utilize this information to boost the performance of downstream heterogeneous graph mining tasks. In terms of generating high-quality and realistic heterogeneous graphs, it is also inevitable to consider modeling the higher-order structural information. However, previous works either are designed for homogeneous graph generation [2, 22] that neglected the importance of the higher-order structural information or fail to consider integrating the higher-order structures into the overall generation block [28]. The distributions of these higher-order graph structures are also hard to capture in heterogeneous graphs, bringing more challenges to effective heterogeneous graph generation. 3) Difficulties in preserving heterogeneous global information. Meta-paths are also well-recognized to play a fundamental role in preserving the global patterns of heterogeneous graphs [11]. For example, the ratio of different node types, and edge types, and their meta-paths are apparently different between the citation networks of the computer system domain and the data mining domain, as shown in Figure 1(a). It is essential to preserve the global distribution of meta-path patterns during heterogeneous graph generation, which is again extremely difficult as it is entangled with the preservation of node type ratios, edge type ratios, and graph topological patterns.

In coping with these challenges, we introduce an end-to-end graph generative framework, namely Heterogeneous Graph Generation (HGEN), whose goal is to generate novel heterogeneous graphs by preserving all the complex local semantic and heterogeneous global property through directly modeling the distribution of meta-paths in observed heterogeneous graphs. Particularly, HGEN learns a joint distribution of the random walks and the associated meta-paths from the observed heterogeneous graphs in order to capture the local semantic distribution. In order to tackle the second difficulty, we extend the meta-path-based generator in HGEN and make it capable of characterizing the higher-order structural distribution via directly modeling and generating network motifs. On top of that, we encode heterogeneous higher-order structural information into nodes via embedding learning and use it to guide the generation of meta-paths and network motifs that form different high-order heterogeneous structures. Finally, to tackle the third challenge, we develop a novel heterogeneous graph assembly method, which is theoretically proved to preserve the global heterogeneous graph patterns in node types, edge types, and meta-paths.

We conclude our major contributions as follows:

• Problem Formulation. We propose to formulate a new paradigm of heterogeneous graph generation, which can effectively identify and resolve its unique challenges in preserving various heterogeneous graph properties.

• Framework Design. We propose an end-to-end generative framework for heterogeneous graph generation. The proposed framework can effectively learn the underlying distribution of heterogeneous graphs. It generates heterogeneous graphs with ensuring the preservation of various heterogeneous graph properties.

• Model Extension. We further extend our proposed model to leverage network motifs to capture more intrinsic higher-order structural information as well as multiple meta relations on edges. We also adapt the proposed graph assembler to adaptively assemble novel graphs by various generated instances.

• Evaluation. We conduct extensive experiments on both synthetic and real-world heterogeneous graphs. Compared with state-of-the-art baselines, HGEN achieves competitive results in preserving most of the static graph properties. In addition, HGEN is shown to be capable of generating realistic heterogeneous graphs by preserving important meta-path information.

2. Related Work

2.1. Graph Generation.

Generative models for graphs have a rich history due to the wide range of applications in different domains, such as link prediction [22, 23], protein structure analysis [29], and information diffusion analysis in social networks [30]. Traditional graph generation methods (e.g., random graphs, stochastic block models, and Bayesian network models) fail to model complex dependencies in our real-world scenarios. In addition, they cannot effectively preserve the statistical properties of the observed graphs. In the last few years, there has been a surge in research focusing on deep graph generation. According to [9], the current deep graph generation can be divided into two categories: sequential-based and one-shot-based. Sequential-based graph generation methods [1, 2, 22] autoregressively generate the nodes and edges with the LSTM model. However, the sequential-based generation (e.g., GraphRNN [2]) is limited in following a fixed node/edge permutation order, which greatly loses the generation flexibility and model scalability. On the other hand, one-shot-based generation methods [22, 23, 28, 29, 31–33] try to build a probabilistic graph model based on the matrix representation that can generate graph topology as well as node/edge attributes in a one-shot, but most of them cannot easily be applied in large graphs due to the large time complexity. For example, GraphVAE [23] is a new and first-of-its-kind variational autoencoder for whole graph generation, though it typically only handles very small graphs and cannot scale well to large graphs in both memory and runtime. NetGAN [22] follows the GAN model [34] and uses a generator to generate synthetic random walks while discriminating synthetic walks from real random walks sampled from a real graph. Finally, multi-attributed graph generation [2, 21, 35, 36] aims at generating homogeneous graphs by preserving node/edge attributes. Instead, the key patterns of heterogeneous graphs are the higher-order local semantics reflected by the combinatorial of the types of nodes and edges, which cannot be captured by methods for homogeneous graphs.

2.2. Heterogeneous Network Motif (Meta-graph)

Compared to the commonly-adopted homogeneous graph, the heterogeneous graph carries much richer semantic information and has therefore gained much attention in recent literature [37]. The concept of meta-paths in a heterogeneous graph [11, 13] is one of the most important concepts proposed to capture numerous semantic relationships across multiple types of objects systematically. Compared to the commonly adopted heterogeneous meta-paths, heterogeneous Network Motifs (also known as meta-graph) [38] are proposed to capture more complex structural information in heterogeneous graphs. Specifically, a meta-graph is a special directed acyclic graph containing at least two embedded metapaths, such as a DAG containing as shown in Figure 1(b), where the higher-order structure one author may publish two papers in a venue contains two meta-paths of author-paper-venue. Network motifs have served as a building block for learning latent embeddings that contain higher-order relationships in a graph [39]. However, in terms of graph generation, graph generative models are successful at retaining pairwise associations in the underlying networks but often fail to capture higher-order connectivity patterns known as network motifs. To date, one attempt [40] leverages network motifs as the basic unit to generate homogeneous graphs. However, this method only learns the structural distribution but fails to capture the meta-relation within the network motifs.

3. Problem Formulation

A heterogeneous graph [10, 37] is a graph $𝒢=\{𝒱,ℰ\}$ with multiple types of objects and relations. $𝒱$ is the set of objects (i.e., nodes), where each node $v_i\in 𝒱$ is associated with a node type $o=\varphi \left(v_i\right).ℰ\subseteq 𝒱\times 𝒱$ is the set of edges, where each edge $e_{ij}\in ℰ$ is associated with a relation type $l=\psi \left(e_{ij}\right)$. All notations are summarized in Table 1.

Table 1:

Description of Important Notations

Notations	Descriptions
$𝒢=\{𝒱,ℰ\}$	A heterogeneous graph $𝒢$ with node set $𝒱$ and edge set $ℰ=𝒱\times 𝒱$
$o=\varphi \left(v_i\right)$	Each node $v_i\in 𝒱$ is associated with a node type $o=\varphi \left(v_i\right)$
$l=\varphi \left(e_{ij}\right)$	Each edge $e_{ij}\in ℰ$ is associated with a relation type $l=\psi \left(e_{ij}\right)$
$(\mathbf{v},\mathbf{o})$	A heterogeneous walk that consists of a random walk $\left(v_1,v_2,\dots ,v_i,\dots \right)$ on $𝒢$ and an associated meta-path $\left(\left(o_1,o_2,\dots ,o_n\right),\left(l_1,l_2,\dots ,l_{n-1}\right)\right)$ of $\mathbf{v}$
$\stackrel{\mathbf{ˆ}}{\mathbf{v}},\stackrel{\mathbf{ˆ}}{\mathbf{o}}$	Generated heterogeneous walk
$S$	Symmetric adjacency matrix with size $𝒱\times 𝒱$ to record the sampled edge frequency

In the study of heterogeneous graphs, the concepts of meta-paths are widely considered as cornerstones and adopted to systematically capture numerous semantic relationships across multiple types of objects, which are defined as a path over the graph [10, 13]. Hence meta-paths are indispensable to be considered as basic units for heterogeneous graph generation. Concretely, a meta-path $\mathbf{o}$ is defined as a sequence of object types and edge types $\mathbf{o}=\left(\left(o_1,o_2,\dots ,o_n\right),\left(l_1,l_2,\dots ,l_{n-1}\right)\right)=o_1\stackrel{l_1}{\to }{o}_2\stackrel{l_2}{\to }\dots \stackrel{l_{n-1}}{\to }{o}_n$, where each $o_i$ and $l_j$ are node type and edge type in the sequence, respectively. Each meta-path captures the rich semantic information between its two ends $o_1$ and $o_n$. In heterogeneous graphs, the local semantic information is carried on each of walks $\mathbf{v}=\left(v_0,v_1,\dots ,v_n\right)$ and its associated meta-path $\mathbf{o}$. We again take Figure 1(c) as an example, there exist two meta-paths between papers: (Paper, Author, Paper) and (Paper, Venue, Paper). The utilization of different meta-paths allow the heterogeneous graph to contain rich topological and semantics among diverse objects, which has been shown beneficial to many real-world graph mining applications [10, 16, 17].

With the preliminary notion of the heterogeneous graph, we formalize the heterogeneous graph generation problem as follows:

Problem 1 (Heterogeneous Graph Generation)

The goal of the heterogeneous graph generation is to learn a distribution $p_{\text{data}}\left(𝒢\right)$ from the observed heterogeneous graphs such that a new graph $\widehat{𝒢}$ can be obtained by sampling $\widehat{𝒢}~p_{\text{data }}\left(𝒢\right)$.

Challenge 1 (Difficulties in modeling the complex local semantic information.)

Although the existence of meta-paths allows heterogeneous graph to characterize the combinatorial of node types and edge types, it is unclear how to model their distributions and generatively assemble them into heterogeneous graphs.

Challenge 2 (Difficulties in characterizing the heterogeneous structural patterns.)

The local structural patterns in heterogeneous graphs are often expressed in higher-order proximity among the nodes and edges (e.g., triangles, orbits, and other higher-order structures). Such a higher-order local structure may fuse multiple walks under one or more meta-paths with richer semantic information, yet brings more difficulties in learning its distribution.

Challenge 3 (Difficulties in capturing heterogeneous global meta-path information.)

Meta-paths indeed play a significant role in preserving the global patterns of heterogeneous graphs. In heterogeneous graph generation, it is important yet challenging to preserve the global distribution of meta-path patterns since the distribution of meta-path patterns often involves node type ratios, edge type ratios, and graph topological patterns.

4. Heterogeneous Graph Generation

To address the above challenges, we propose a new heterogeneous graph generation framework, named HGEN. To address the first and second challenge, we propose a heterogeneous walk generator in Section 4.1 to jointly learn the distribution of local walks and the associated meta-paths so that both heterogeneous topological and local semantic information can be well captured. To overcome the second challenge, we leverage the heterogeneous node embedding to make the generator be aware of any potential higher-order structures that each node may be involved with. Finally, for the third challenge, we propose a novel heterogeneous graph assembler in Section 4.3, which can construct new heterogeneous graphs by capturing the global heterogeneous property, namely different meta-path ratios. We further prove that the global heterogeneous property can be well-preserved through our Theorem 1 introduced in Section 4.4.

4.1. Heterogeneous Walk Generator

In the observed graph $𝒢$, a heterogeneous walk is defined as a tuple that consists of two components: a walk $\mathbf{v}$ and an associated meta-path o. The proposed heterogeneous walk generator $G$ is defined as a probabilistic sequential learning model to generate synthetic heterogeneous walks: $(\stackrel{\mathbf{ˆ}}{\mathbf{v}},\stackrel{\mathbf{ˆ}}{\mathbf{o}})=\left(\left({\hat{v}}_1,{\hat{v}}_2,\dots ,{\hat{v}}_n\right),\left(\left({\hat{o}}_1,{\hat{o}}_2,\dots ,{\hat{o}}_n\right),\left({\hat{l}}_1,{\hat{l}}_2,\dots ,{\hat{l}}_{n-1}\right)\right)\right)$, where the $\stackrel{\mathbf{ˆ}}{\mathbf{v}}$ and $\stackrel{\mathbf{ˆ}}{\mathbf{o}}$ are denoted as the generated walk and associated meta-path, respectively. We use ${\hat{v}}_i,{\hat{o}}_i$, and ${\hat{l}}_i$ to denote each of the generated node, node type, and edge type in $(\stackrel{\mathbf{ˆ}}{\mathbf{v}},\stackrel{\mathbf{ˆ}}{\mathbf{o}})$, respectively. Figure 2(a) illustratively summarizes the whole generative process of each synthetic heterogeneous walk.

Fig. 2:

The illustration of the heterogeneous walks generation in HGEN.

Heterogeneous Walk Generation.

We model $G$ as a sequential learning process based on a recurrent architecture, and each unit $f_{\theta }$ in the sequential model is parameterized by $\theta$ so that it can generate a node type $\hat{o}$ and a corresponding node $\hat{v}$ that belongs to this node type in a hierarchical manner. Precisely, the node type $\hat{o}$ is determined based on the previously generated sequence, and the node $\hat{v}$ is then coherently determined by the generated node type as well as the generated sequence. Both generated node type $\hat{o}$ and node $\hat{v}$ together provide information for the generation of the next node type and node instance.

Specifically, at each recurrent block (i.e., time step) $t,f_{\theta }$ produces two outputs $\left({\mathit{m}}_t,{\mathit{h}}_t\right)$, where the ${\mathit{m}}_t$ is the current memory state and the ${\mathit{h}}_t$ is a latent probabilistic distribution (i.e., hidden output of $f_{\theta }$) denoting the information carried from previous time steps. We first sample the node type ${\hat{o}}_t~g_o\left({\mathit{h}}_t\right)$ based on the probability distribution ${\mathit{h}}_t$, where the $g_o(\cdot )$ is a node type decoding function. We then sample the node ${\hat{v}}_t$ by a node decoding function ${\hat{v}}_t~g_v\left({\mathit{h}}_t,{\hat{o}}_t\right)$ that takes ${\mathit{h}}_t$ and ${\hat{o}}_t$ as inputs. Lastly, the generated node type ${\hat{o}}_t$ and node ${\mathit{h}}_t$ are fused by a heterogeneous node encoding function $g_c\left({\hat{o}}_t,{\hat{v}}_t\right)$, which then serves as the input of next recurrent block.

Heterogeneous Node Sampling.

To overcome the second challenge, we cannot uniformly sample ${\hat{v}}_t$ based on the node type ${\hat{o}}_t$ because such a way may cause the neglection of (1) node structural distribution and (2) node semantic distribution. For example, we may observe an author always tends to cite a paper with high citation (namely, high node degree of this paper node). Then such distribution needs to be modeled with structural information. On the other hand, we may observe a data mining paper is unlikely to cite a computer system paper, and we may also need to characterize this tendency in the distribution. Both of the above distributions cannot be tackled by uniformly sampling. Therefore, to tackle this challenge, since latent node embedding could encode both topological and semantic information into the node, we propose to calculate a latent embedding ${\tilde{v}}_t$ of the next node $v_t$, then we select with a higher probability the closer embedding among all the embeddings that belong to node type ${\hat{o}}_t$ so that the next node $v_t$ can be determined by the sampled embedding.

More specifically, we first calculate the latent node embedding ${\tilde{v}}_t$ based on the sampled node type ${\hat{o}}_t$ by a simple linear transformation. We then calculated the distance between ${\tilde{v}}_t$ and other node embedding ${\tilde{v}}_i^{\left({\hat{o}}_t\right)}$, meaning any node ${\tilde{v}}_i$ belonging to the sampled node type ${\hat{o}}_t$. In this case, given a total number of $k$ embeddings that belong to the type ${\hat{o}}_t$, the next node ${\hat{v}}_t$ can be sampled from a multinomial distribution:

$${\hat{v}}_t~\text{Multi}\left({\tilde{v}}_1^{\left({\hat{o}}_t\right)},{\tilde{v}}_2^{\left({\hat{o}}_t\right)},\dots ,{\tilde{v}}_k^{\left({\hat{o}}_t\right)};p_1,p_2,\dots ,p_k\right),$$

where each $p_i=-{∥d\left({\tilde{v}}_t,{\tilde{v}}_i^{\left({\hat{o}}_t\right)}\right)∥}^2$ and $d(\cdot ,\cdot )$ is a distance metric such as Euclidean distance. Note that the node embedding ${\tilde{v}}_i^{\left({\hat{o}}_t\right)}$ can be obtained from a conventional heterogeneous node embedding technique such as [14].

In order to generate a variable-length heterogeneous walk, we incorporate a end-of-sequence token as an additional node type so that the heterogeneous walk generator stops when the sampled node type is the token at any steps. Therefore, the proposed generator is able to produce variable-length heterogeneous walks. Finally, the edge type $l_t$ can be predicted by a simple edge decoding function $g_e\left({\hat{o}}_t,{\hat{v}}_t,{\hat{o}}_{t-1},{\hat{v}}_{t-1}\right)$ that takes its two end nodes ${\hat{v}}_{t-1}$ and ${\hat{v}}_t$ as well as their node types ${\hat{o}}_{t-1}$ and ${\hat{o}}_t$ as inputs. In all, we summarize the overall generative process as follows:

$$\begin{gathered} {\mathit{a}}_0=0,{\mathit{m}}_0=f_0\left(\mathit{z}\right),\mathit{z}~𝒩\left(\mathrm{0,1}\right) \\ {\mathit{a}}_1=g_c\left({\hat{o}}_1,{\hat{v}}_1\right),{\hat{v}}_1~g_v\left({\mathit{h}}_1,{\hat{o}}_1\right),{\hat{o}}_1~g_o\left({\mathit{h}}_1\right),\left({\mathit{m}}_1,{\mathit{h}}_1\right)=f_{\theta }\left({\mathit{m}}_0,{\mathit{a}}_0\right) \\ {\mathit{a}}_2=g_c\left({\hat{o}}_2,{\hat{v}}_2\right),{\hat{v}}_2~g_v\left({\mathit{h}}_2,{\hat{o}}_2\right),{\hat{o}}_2~g_o\left({\mathit{h}}_2\right),\left({\mathit{m}}_2,{\mathit{h}}_2\right)=f_{\theta }\left({\mathit{m}}_1,{\mathit{a}}_1\right) \\ {\hat{l}}_1=g_e\left({\hat{o}}_2,{\hat{v}}_2,{\hat{o}}_1,{\hat{v}}_1\right) \\ \cdots \\ {\hat{v}}_n~g_v\left({\mathit{h}}_n,{\hat{o}}_n\right),{\hat{o}}_n~g_o\left({\mathit{h}}_n\right),\left({\mathit{m}}_n,{\mathit{h}}_n\right)=f_{\theta }\left({\mathit{m}}_{n-1},{\mathit{a}}_{n-1}\right) \\ {\hat{l}}_{n-1}=g_e\left({\hat{o}}_n,{\hat{v}}_n,{\hat{o}}_{n-1},{\hat{v}}_{n-1}\right) \end{gathered}$$

In this work, we utilize LSTM as the recurrent architecture, and $f_{\theta }$ becomes a single LSTM unit. To initialize the whole generative process, $G$ takes a random noise $\mathit{z}$ as input, which is drawn from a standard Gaussian distribution. Additionally, for the node type decoding function $g_o(\cdot )$, we apply the Gumbelsoftmax trick [41] in $g_o(\cdot )$ to make the whole sampling differentiable. Finally, in most of the real-world scenarios, the edge type $l_t$ can be determined by the types of its two end nodes ${\hat{o}}_t$ and ${\hat{o}}_{t-1}$ if there does not exist multi-typed relations between two node types. In this case, the heterogeneous walk generator can be simplified only to generate node sequences and associated node types.

4.2. Extension of Heterogeneous Motif Generator.

In the previous section, the proposed heterogeneous walk generator can well characterize pairwise relationships within the heterogeneous graph and associated heterogeneous graph statistics via meta-paths; however, higher-order relationships (aka. heterogeneous network motifs) in a heterogeneous graph are fundamental for our understanding of the network behavior and function.

Definition 1 (Heterogeneous Network Motifs) A Heterogeneous Network Motif (Meta Graph) $ℳ$ is a directed acyclic graph (DAG) with a single source node $v_s$ (i.e., with in-degree 0) and a single target node $v_t$ (i.e., with out-degree 0) defined on a heterogeneous graph $𝒢=\{𝒱,ℰ\}$. Then we define a Heterogeneous Network Motif as $ℳ=\left({𝒱}_{ℳ},{ℰ}_{ℳ},v_s,v_t\right)$, where ${𝒱}_{ℳ}\subseteq 𝒱$ and ${ℰ}_{ℳ}\subseteq 𝒱$.

As we emphasized in Section 3, meta-path is the natural way to represent local semantics in heterogeneous networks; however, meta-path may not be the best way to characterize the rich semantics, especially semantics encoded in higher-order structures. As can be clearly seen in Figure 4, separate meth-paths (i.e., author-paper-topic and author-paper-venue) can form an orbit structure, which cannot be simply described by meta-paths. Current heterogeneous graph generation methods either rely on meta-paths as the basic generation unit [28] or completely ignore the rich semantics encoded in heterogeneous meta-structures [20], which is a major shortcoming for applications that aim to generate heterogeneous graphs that realistically mimic real-world heterogeneous networks or predict unobserved heterogeneous higher-order structures.

Fig. 4:

Various graph motifs (meta-graphs) in academic heterogeneous network: one author publishes two paper that one paper cites the other one; one author publishes two papers that related to the same topic; and one author publishes two co-cited papers at one venue.

In order to make our algorithm better preserve the higher-order structural distribution in the generated graph, other than utilizing heterogeneous node embedding, we generalize the proposed heterogeneous walk generator to be able to generate heterogeneous network motifs. While a complete enumeration of the network motifs present in a large-scale heterogeneous network is computationally prohibitive, we instead focus on three motif structures (e.g., triangle, orbit, overlapped triangle) as visualized in Figure 4.

Since graph motifs contain DAG structure, we may not trivially generate them as varying-length sequences and leverage the end-of-sequence token to indicate the stop. Instead, we propose to sample from a learnable logit $\tau$, where each ${\tau }_i\in \tau$ represents the probability of the motif chosen to be generated and $\parallel \tau \parallel =1$. Note that we initialize ${\tau }_0\in \tau$ to be the probability of choosing meta-path to generate so that the motif-based generation model can be combined training with the meta-path-based generator. We provided the updated model overview as follows.

4.3. Heterogeneous Generator Training and Utilization

In the following, we will introduce how to train the above-mentioned generator and how to use the heterogeneous walks and motifs generated by it to construct heterogeneous graphs. Since we extend our framework to be able to generate both meta-paths and motifs, we refer the generated meta-paths and motifs to heterogeneous instances for the sake of simplicity in the following context. Concretely, we utilize a heterogeneous discriminator $D$ to distinguish between real and fake heterogeneous instances, where the real instances are uniformly sampled from the observed graph. We then propose a heterogeneous graph assembler to construct new graphs based on the sampled heterogeneous instances. More details are presented as follows.

We first introduce the overall objective function of the Wasserstein heterogeneous GAN [34], which is written as:

$${ℒ}_{\text{HGEN}}=\text{max }{\mathbb{E}}_{(\mathbf{o},\mathbf{v})~p\left(𝒢\right)}\left[D_o\left(\mathbf{o}\right)+D_v\left(\mathbf{v}\right)\right]-{\mathbb{E}}_{z~p\left(z\right)}\left[D_o\left(\stackrel{\mathbf{ˆ}}{\mathbf{o}}\right)+D_v\left(\stackrel{\mathbf{ˆ}}{\mathbf{v}}\right)\right],s.t.G\left(z\right)=(\stackrel{\mathbf{ˆ}}{\mathbf{o}},\stackrel{\mathbf{ˆ}}{\mathbf{v}}),$$ (1)

where $\mathbf{v}$ and $\mathbf{o}$ are the random walk/motif and associated meta-path/metagraph, respectively, directly sampled from the observed heterogeneous graph $𝒢$. They are the real data for training our heterogeneous generator $G$. Specifically, given an observed heterogeneous graph $𝒢=\{𝒱,ℰ\}$, we utilize random-walkbased method to uniformly sample a set of random walks $\left\{{\mathbf{v}}_1,{\mathbf{v}}_2,\dots \right\}$, where each ${\mathbf{v}}_i$ is a node sequence s.t. ${\mathbf{v}}_i=\left(v_1,v_2,\dots ,v_n\right)$. In addition, we extract the meta information ${\mathbf{o}}_i=\left(\left(o_1,o_2,\dots ,o_n\right),\left(l_1,l_2,l_{n-1}\right)\right)$ from each ${\mathbf{v}}_i$.

The heterogeneous discriminator $D$ in Equation (1) is designed as a parallel recurrent architecture in order to individually distinguish whether each unit in the heterogeneous component are valid or not. Specifically, at each recurrent block (i.e., each step) $t$, the discriminator $D$ takes two inputs: the generated node type ${\hat{o}}_t$ and node index ${\hat{v}}_t$, each of which is fed into an individual recurrent unit. After processing both sequences, the discriminator returns a single score $D_v\left(\mathbf{v}\right)+D_o\left(\mathbf{o}\right)$ that represents the probability of the heterogeneous component being real.

4.3.1. Heterogeneous Graph Assembler

To assemble a heterogeneous graph from the generated heterogeneous instances, we further propose a novel stratified heterogeneous edge sampling strategy to achieve the following steps: 1) it first samples a node ${\hat{v}}_i$ and its type ${\hat{o}}_i$ from all of the generated heterogeneous walks; 2) based on the node type ${\hat{o}}_i$, we then sample a meta-path that starts with ${\hat{o}}_i;$ 3) we iteratively sample the next node ${\hat{v}}_{i+1}$ in the sampled meta-path if both of the node type ${\hat{o}}_{i+1}$ and edge type ${\hat{l}}_i$ fits the meta-path pattern.

More specifically, the generator $G$ firstly produces a sufficient number of heterogeneous walks as shown in Figure 3(a). We then construct an symmetric adjacency matrix $S$ with size $\left|𝒱\right|\times \left|𝒱\right|$ to record the count of edges observed from the sampled heterogeneous walks in each entry $S_{ij}$, where the $\left|𝒱\right|$ is the size of the node set. Next, we collect all of the meta-path patterns generated by the generated heterogeneous walks, as shown in Figure 3(b–c). For the first step of the stratified ${\hat{v}}_i$ and its type type ${\hat{o}}_i$ based on the node degree distribution $\frac{{\sum }_j{S}_{ij}}{\left|𝒱\right|}$. For the second step, among all the meta-paths $\left\{{\mathbf{o}}_1^{\left(f\right)},{\mathbf{o}}_2^{\left(f\right)},\dots \right\}$ that start with the node type ${\hat{o}}_i$, we sample a meta-path ${\mathbf{o}}_i^{\left(f\right)}$ based on the probability $\frac{c\left({\mathbf{o}}_i^{\left(f\right)}\right)}{T^{{\hat{o}}_i}}$, where $T^{{\hat{o}}_i}$ is the total count of generated meta-paths that starts with node type ${\hat{o}}_i$ and $c\left({\mathbf{o}}_i^{\left(f\right)}\right)$ is the count of meta-path pattern ${\mathbf{o}}_i^{\left(f\right)}$. For the third step, by following this meta-path pattern ${\mathbf{o}}_r=\left(o_1,o_2,\dots ,o_n\right)$, we iteratively sample all the nodes whose node types are regulated by the the meta-path. Precisely, we sample the next node $v_j$ by sampling all the neighbors of the current node $v_i$ with the probability $p_{v_i{v}_j}=\left(S_{ij}\right)/\left({\sum }_s{S}_{is}\right)$ such that all the nodes $v_s$ belong to the specific node type $o_j$ following the meta-path ${\mathbf{o}}_i^{\left(f\right)}$. The sampled node sequence ${\mathbf{v}}_r=\left(v_0,v_1,\dots \right)$ is then added to the score matrix $S$. We continue the stratified heterogeneous edge sampling strategy until the desired amount of edges is reached. The final assembled graph is visualized in Figure 3 (d).

Fig. 3:

The process of heterogeneous graph assembler.

Extension of Heterogeneous Graph Assembler with Motif Consideration.

To assemble a heterogeneous graph from the generated heterogeneous instances, we further extend our stratified heterogeneous graph assembler and leverage the learned logit $\tau$ in order to make the assembler generate heterogeneous graph that has the closer higher-order structural distribution. Specifically, after the generator $G$ produces a sufficient number of heterogeneous instances, we leverage the learned $\tau$ to firstly sample the exact heterogeneous instance pattern (i.e., walk, triangle, or orbit). After collecting such a pattern, we then follow the aforementioned stratified heterogeneous edge sampling strategy to sample exact nodes under the specific pattern. This strategy allows us to more closely model the local semantic, higher-order, and global distribution if the learned $\tau$ can correctly characterize the ratio of different heterogeneous component patterns in the observed heterogeneous graph.

4.3.2. Complexity Analysis.

The computational complexity of HGEN is $O(W\cdot L)$, where $W$ is the weights of a single LSTM unit, and $L$ is the length of the generated heterogeneous instances. However, the length of our proposed heterogeneous walk is considerably small $(1\le L\le 3)$ while the walk length in other random-walk-based graph generative method [22] is $(\ge 16)$. For auto-regressive graph generation models [2, 3], the time complexities are at least $O\left(|𝒱{|}^2\cdot W\right)$, where $\left|𝒱\right|$ is the cardinality of the node set. They convert graph as a long sequence by performing a large number of breadth-first-search (BFS) enumerations for each graph. Additionally, HGEN also has linear complexity in graph assembly, it only needs to run the trained model $T_s$ times to sample heterogeneous walks for constructing the score matrix $S$. To sum up, the overall complexity of HGEN can be reduced to $O\left(W+T_s\right)$, which makes our proposed model highly efficient for handling large graphs, since the overall process is not sensitive to the number of nodes at all.

4.4. Meta-path Information Preservation Analysis

As we discussed in Sec. 3, it is significant to preserve the meta-path information in our generated graph. Taking Fig. 6 as an example, although both graphs have exactly the same structure, they are still regarded as two different heterogeneous graphs since their meta-path distributions are different. Given the importance of the meta-path information in heterogeneous graph generation, we further show that our framework can successfully preserve this meta-path information as proved in Theorem 1.

Fig. 6:

Example of two heterogeneous graphs with different semantic information: the observed meta-path patterns are different, although the node and edge distribution are the same between two graphs. Specifically, since we do not observe a direct link between (author, venue) and (paper, paper) in the observed graph Figure 6(a). It is not accurate for the generated graph Figure 6(b) that generate such links.

Theorem 1 The distribution of meta-path patterns ${\stackrel{-}{𝒪}}^{\left(r\right)}$ of the generated heterogeneous graph equals the distribution of meta-path patterns $\stackrel{-}{𝒪}$ in the observed heterogeneous graph, namely $p\left({\stackrel{-}{𝒪}}^{\left(r\right)}\right)=p\left(\stackrel{-}{𝒪}\right)$.

Proof. We will prove that the ratio of the meta-path patterns can be preserved in three steps: 1) the ratio of different meta-path patterns can be preserved during the sampling procedure; 2) the ratio of generated meta-path patterns can be preserved during the generation procedure; 3) the meta-path patterns can be preserved during the graph assembling procedure.

Meta-path Ratio Preservation in Sampling.

Let $\stackrel{-}{𝒪}=\left({\stackrel{-}{\mathbf{o}}}_1,{\stackrel{-}{\mathbf{o}}}_2,\dots \right)$ be the collection of meta-paths obtained from the observed heterogeneous graph $𝒢$, each ${\stackrel{-}{\mathbf{o}}}_i$ is a meta-path in one-hot format ${\stackrel{-}{\mathbf{o}}}_i\in \{\mathrm{0,1}{\}}^{1\times R}$, where the $R$ is the total number of different meta-path patterns. ${\stackrel{-}{𝒪}}^{\left(\tau \right)}=\left({\stackrel{-}{\mathbf{o}}}_1^{\left(\tau \right)},{\stackrel{-}{\mathbf{o}}}_2^{\left(\tau \right)},\dots ,{\stackrel{-}{\mathbf{o}}}_K^{\left(\tau \right)}\right)$ is the sequence of sampled meta-paths with sampling size $K$, where each meta-path ${\stackrel{-}{\mathbf{o}}}_j^{\left(\tau \right)}\in \{\mathrm{0,1}{\}}^{1\times R}$ is drawn independent and identically distributed (i.i.d) from $\stackrel{-}{𝒪}$.

Suppose that $\mathit{\mu }={\left[{\mu }_1,{\mu }_2,\dots ,{\mu }_R\right]}^T$ denotes the probability of each individual meta-path pattern in $\stackrel{-}{𝒪}$, it is obvious that $\mathbb{E}\left[{\stackrel{-}{\mathbf{o}}}_i\mid \mathit{\mu }\right]={\sum }_{{\stackrel{-}{\mathbf{o}}}_i}p\left({\stackrel{-}{\mathbf{o}}}_i\mid \mathit{\mu }\right){\stackrel{-}{\mathbf{o}}}_i={\left[{\mu }_1,{\mu }_2,\dots ,{\mu }_R\right]}^T=\mathit{\mu }$. Now consider the total $K$ observations ${\stackrel{-}{𝒪}}^{\left(\tau \right)}=\left({\stackrel{-}{\mathbf{o}}}_1^{\left(\tau \right)},{\stackrel{-}{\mathbf{o}}}_2^{\left(\tau \right)},\dots ,{\stackrel{-}{\mathbf{o}}}_K^{\left(\tau \right)}\right)$, the corresponding likelihood function takes the form:

$$p\left({\stackrel{-}{𝒪}}^{\left(\tau \right)}\mid \mathit{\mu }\right)=\prod _i^R\prod _j^K{\mu }_j^{{\stackrel{-}{\mathbf{o}}}_{ij}^{\left(\tau \right)}}=\prod _j^K{\mu }_j^{{\sum }_n{\stackrel{-}{\mathbf{o}}}_{nj}^{\left(\tau \right)}}=\prod _j^K{\mu }_j^{m_j}$$ (2)

We see that the likelihood function depends on the $K$ data points only through the $R$ quantities: $m_j={\sum }_n{\stackrel{-}{\mathbf{o}}}_{nj}^{\left(\tau \right)}$. Since the number of observations of ${\stackrel{-}{\mathbf{o}}}_j^{\left(\tau \right)}$ equals 1, we achieved sufficient statistics for this distribution. Therefore, $p\left({\stackrel{-}{𝒪}}^{\left(\tau \right)}\right)=p\left(\stackrel{-}{𝒪}\right)$ can be proved.

Meta-path Ratio Preservation in Generation.

Since we have proved the meta-path ratio can be preserved during the sampling, the next step is to show that the distribution of generated meta-paths $p\left({\stackrel{-}{𝒪}}^{\left(g\right)}\right)$ is equal to $p\left({\stackrel{-}{𝒪}}^{\left(\tau \right)}\right)$. Proving $p\left({\stackrel{-}{𝒪}}^{\left(g\right)}\right)=p\left({\stackrel{-}{𝒪}}^{\left(\tau \right)}\right)$ is equivalent to prove whether $p_{\mathit{\text{data}}}=p_g$ in the GAN setting. As being proved in the works of GANs and their variants [34, 42], it showed that the objective function of the generator $G$ is equivalent to optimize the distribution distance between $p_{\mathit{\text{data}}}$ and $p_g$ if the discriminator $D$ is optimal. Therefore, global optimality of $p_g=p_{\mathit{\text{data}}}$ can be achieved if both generator $G$ and discriminator $D$ have enough capability. Therefore, $p\left({\stackrel{-}{𝒪}}^{\left(g\right)}\right)=p\left({\stackrel{-}{𝒪}}^{\left(\tau \right)}\right)$ if both $G$ and $D$ are optimal in our framework.

Meta-path Ratio Preservation in Assembling.

Finally, we show that our graph assembling method can also preserve the meta-path ratio from the generated data ${\stackrel{-}{𝒪}}^{\left(g\right)}$ such that $p\left({\stackrel{-}{𝒪}}^{\left(g\right)}\right)=p\left({\stackrel{-}{𝒪}}^{\left(r\right)}\right)$. As discussed in Sec. 4.3, the new graph $\widehat{𝒢}$ is directly assembled by meta-paths $\left({\stackrel{-}{\mathbf{o}}}_1^{\left(g\right)},{\stackrel{-}{\mathbf{o}}}_2^{\left(g\right)},\dots ,{\stackrel{-}{\mathbf{o}}}_Q^{\left(g\right)}\right)$ that are sampled i.i.d from ${\stackrel{-}{𝒪}}^{\left(g\right)}$ with sampling size $Q$, which is exactly the reverse procedure of Eq. (2).

Therefore, if both generator $G$ and discriminator $D$ are optimal, the multinomial distribution $p\left(\stackrel{-}{𝒪}\right)$ of distinct meta-path patterns can be preserved in all three steps of our generation framework. □

5. Experiment

In this section, we compare HGEN to the adaption of closest state-of-the-art baselines, demonstrating its effectiveness in generating realistic heterogeneous graphs in diverse settings. The code and dataset can be found at: https://github.com/lingchen0331/HGEN.

5.1. Data

We perform experiments on three synthetic heterogeneous graph datasets and three real-world heterogeneous graph datasets. We summarize the statistics of datasets in Table 2.

Table 2:

Dataset Overview.

	# of Nodes	# of Edges	Average Degree	# of Node Types	# of Edge Types
Syn100	100	490	9.8	3	6
Syn200	200	1,090	10.9	3	6
Syn500	500	2,987	11.95	3	6
SynMulti	300	1,352	12.13	3	8
PubMed	1,565	13,532	17.29	4	10
IMDB	1,653	4,267	5.432	4	4
DBLP	11,240	47,885	8.52	4	3

Synthetic Datasets.

We synthesis random heterogeneous graphs of different sizes through the combination of $N$ overlapping homogeneous graphs, where the overlap is accomplished by node sharing. We generate three random heterogeneous graphs (named as ${\text{Syn}}_{100},{\text{Syn}}_{200}$, and ${\text{Syn}}_{500}$) with node size 100, 200, and 500, respectively. The number of node types in each of the synthetic heterogeneous graph is 3. In addition, we sample a random heterogeneous graph ${\text{Syn}}_{\mathit{\text{Multi}}}$ with node size 300 that contains multiple edge types between two nodes.

Real-world Datasets.

We also employ three large-scale real-world heterogeneous graph datasets in our experiment.

• PubMed. This dataset consists of four classes of nodes: Gene (G), Disease (D), Chemical (C), and Species (S). We construct a sub-graph that relates to all Chemical nodes labeled in [10]. There are 1, 565 nodes and 13, 532 edges.

• IMDB. This movie-related heterogeneous graph is adopted from [17], which contains three node types: Director (D), Actor (A), Movie (M), and Genre (G). We construct a subgraph that contains all the movies with a score $\ge 7.5$. This graph contains 1, 653 nodes and 4, 267 edges.

• DBLP. This heterogeneous graph adopted from [17] contains Paper (P), Author (A), Venue (V), and Term (T) as node types. We sample a sub-graph that is related to five computer science venues: $KDD,\mathit{\text{WSDM}},WWW,\mathit{\text{ICDM}}$, and $\mathit{\text{ICML}}$. There are 1, 565 nodes and 47, 885 edges.

5.2. Experiment Setting

In our experiment, we focus on meta-paths with length 1, 2, and 3 as they are the most common ones in heterogeneous graphs [11]. We sample 10 graphs from each of the trained models and report their average results and standard deviation in Table 3. We randomly select 60% of the edges for training, and the remaining graph is used for testing.

Table 3:

Performance evaluation over compared baselines. The Real rows include the values of real graphs, while the rest are the evaluation results of different algorithms. The best performance (the closest to real value) achieved under each metric for a particular dataset is highlighted in bold font. Note that we do not include GraphVAE in datasets with (≥ 300) nodes and GraphRNN in datasets with (≥ 10,000) nodes because the programs return errors.

Graphs	Models	LCC	TC	Clustering Coef.	Powerlaw Coef.	Assortativity	Degree Distribution Dist.	EO Rate	Uniqueness
Syn-100	GraphRNN	78.43 ± 2.23	16.62 ± 5.42	0.002 ± 0.01	1.611 ± 0.09	−0.153 ± 0.07	2.19e–2 ± 3.21e–3	37.21% ± 1.08%	33.09% ± 7.06%
	NetGAN	80.12 ± 3.45	6.79 ± 1.76	0.001 ± 0.00	1.524 ± 0.21	−0.213 ± 0.09	1.33e–2 ± 6.46e–3	8.74% ± 0.82%	94.03% ± 0.49%
	GraphVAE	99.01 ± 0.00	224.81 ± 5.13	0.70 ± 0.04	4.579 ± 0.05	−0.73 ± 0.05	3.71e–1 ± 1.98e–2	11.5% ± 1.09%	65.54% ± 2.98%
	VGAE	48.9 ± 4.63	63.7 ± 46.25	0.184 ± 0.06	1.87 ± 0.10	0.1 ± 0.03	2.23e–1 ± 6.08e–2	3.23% ± 0.09%	51.1% ± 3.04%
	HGEN	81.13 ± 2.42	53.12 ± 3.78	0.079 ± 0.01	1.782 ± 0.01	−0.114 ± 0.03	8.79e–3 ± 3.12e–3	10.2% ± 0.17%	92.97% ± 0.72%
	HGEN-Motif	80.54 ± 3.12	47.34 ± 2.69	0.089 ± 0.01	1.673 ± 0.02	−0.207 ± 0.06	6.28e–2 ± 1.77e–3	13.7% ± 0.17%	89.32% ± 1.32%
	Real	85	36	0.072	1.832	−0.169	N/A	N/A	N/A
Syn-200	GraphRNN	132.76 ± 1.08	2.54 ± 0.77	0.001 ± 0.00	1.603 ± 0.01	−0.05 ± 0.01	5.15e–2 ± 3.07e–3	25.81% ± 2.65%	27.72% ± 3.07%
	NetGAN	153 ± 1.56	2.24 ± 0.35	0.001 ± 0.00	1.579 ± 0.31	−0.008 ± 0.001	6.43e–2 ± 4.2e–3	11.32% ± 0.77%	95.88% ± 3.19%
	GraphVAE	195.43 ± 1.12	51.32 ± 1.01	0.002 ± 0.001	5.377 ± 0.21	−0.75 ± 0.05	5.38e–1 ± 1.7e–2	1.78% ± 0.41%	64.37% ± 2.94%
	VGAE	86.2 ± 16.93	860.4 ± 185.9	0.23 ± 0.04	1.787 ± 0.08	0.2 ± 0.15	8.53e–2 ± 2.14e–2	3.74% ± 0.08%	59.65% ± 1.46%
	HGEN	158.5 ± 2.64	38.5 ± 5.26	0.043 ± 0.01	1.732 ± 0.02	–0.065 ± 0.04	2.25e–2 ± 5.5e–3	4.22% ± 0.67%	96.31% ± 5.11%
	HGEN-Motif	169 ± 4.89	29.28 ± 3.43	0.049 ± 0.01	1.72 ± 0.02	−0.015 ± 0.08	3.73e–3 ± 2.5e-4	8.6% ± 1.3%	93.57% ± 3.61%
	Real	180	28	0.037	1.809	−0.089	N/A	N/A	N/A
Syn-500	GraphRNN	311.59 ± 2.14	11.53 ± 5.57	0.004 ± 0.001	1.862 ± 0.01	1.862 ± 0.002	4.05e–2 ± 1.1e–3	21.87% ± 0.86%	29.54% ± 4.32%
	NetGAN	305.81 ± 14.28	3 ± 1.21	0.001 ± 0.001	1.812 ± 0.07	0.03 ± 0.12	4.83e–2 ± 7.4e–4	6.72% ± 0.13%	93.98% ± 0.21%
	VGAE	97.0 ± 29.24	4346.2 ± 453.62	0.193 ± 0.02	1.77 ± 0.06	−0.022 ± 0.09	2.22e–1 ± 2.4e–2	5.46% ± 1.12%	63.65% ± 3.1%
	HGEN	347.88 ± 7.63	74.88 ± 4.78	0.031 ± 0.01	1.865 ± 0.02	−0.097 ± 0.01	2.81e–2 ± 3.4e–3	1.49% ± 0.11%	95.89% ± 1.18%
	HGEN-Motif	398 ± 7.23	59.63 ± 5.68	0.024 ± 0.03	1.72 ± 0.02	−0.015 ± 0.08	3.73e–3 ± 2.5e-4	10.32% ± 2.5%	89.67% ± 1.32%
	Real	417	8	6.5e–3	1.978	−0.12	N/A	N/A	N/A
Syn-Multi	GraphRNN	219.32 ± 32.65	23.67 ± 13.88	0.003 ± 0.001	0.962 ± 0.02	1.373 ± 0.04	3.77e–2 ± 2.9e–3	36.71% ± 4.66%	42.12% ± 4.99%
	NetGAN	215.48 ± 6.99	23 ± 4.74	0.02 ± 0.001	1.076 ± 0.03	0.67 ± 0.22	3.78e–2 ± 3.5e–3	19.87% ± 0.54%	89.13% ± 0.34%
	VGAE	106.8 ± 37.12	1432.8 ± 283.12	0.087 ± 0.03	1.48 ± 0.09	−0.039 ± 0.12	2.22e–1 ± 2.4e–2	7.93% ± 0.78%	27.78% ± 9.3%
	HGEN	295.82 ± 3.91	35.71 ± 5.76	0.019 ± 0.008	1.381 ± 0.03	−0.055 ± 0.03	1.38e–2 ± 1.3e–3	1.49% ± 0.11%	95.89% ± 1.18%
	HGEN-Motif	282.78 ± 3.79	43.32 ± 2.99	0.023 ± 0.003	1.087 ± 0.05	−0.079 ± 0.04	1.21e–2 ± 2.4e–4	2.88% ± 0.57%	91.32% ± 1.07%
	Real	275	47	0.027	1.243	−0.36	N/A	N/A	N/A
PubMed	GraphRNN	1563.23 ± 32.46	1549.79 ± 33.62	0.01 ± 0.007	1.753 ± 0.04	−0.03 ± 0.01	1.61e–1 ± 3.71e–2	13.41% ± 1.24%	54.62% ± 4.32%
	NetGAN	793.2 ± 41.5	18.3 ± 0.9	0.001 ± 0.00	1.47 ± 0.11	−0.12 ± 0.02	6.69e–2 ± 1.5e–3	4.32% ± 0.54%	78.03% ± 0.19%
	VGAE	347.9 ± 7.03	70,982.2 ± 4,086.53	0.234 ± 0.01	2.48 ± 0.01	−0.466 ± 0.01	1.38e–1 ± 4.8e–3	≈0%	22.87% ± 1.68%
	HGEN	825.6 ± 22.1	1569.3 ± 31.3	0.034 ± 0.003	1.634 ± 0.07	−0.143 ± 0.08	3.92e–2 ± 7.5e–4	0.07% ± 0.01%	93.91% ± 0.12%
	HGEN-Motif	897.3 ± 12.5	1972.3 ± 46.7	0.051 ± 0.002	1.505 ± 0.04	−0.162 ± 0.07	5.21e–2 ± 6.2e–4	0.12% ± 0.03%	94.12% ± 0.33%
	Real	948	2,114	0.068	1.75	−0.208	N/A	N/A	N/A
IMDB	GraphRNN	1425.47 ± 121.5	142.13 ± 5.87	0.179 ± 0.02	2.97 ± 0.05	0.05 ± 0.04	1.98e–1 ± 2.61e–3	9.87% ± 0.51%	21.52% ± 3.31%
	NetGAN	932.5 ± 8.49	0.0 ± 0.0	0.0 ± 0.0	2.08 ± 0.01	−0.25 ± 0.07	1.36e–1 ± 1.89e–3	7.62% ± 0.07%	82.69% ± 1.27%
	VGAE	635.2 ± 4.16	7,752.4 ± 281.32	0.141 ± 0.01	2.02 ± 0.02	−0.49 ± 0.15	1.9e–1 ± 2.33e–3	≈0%	42.71% ± 1.47%
	HGEN	945.2 ± 11.54	26.0 ± 3.28	3.56e–3 ± 0.0	2.16 ± 0.01	−0.19 ± 0.04	4.36e–2 ± 4.25e–4	2.69% ± 0.04%	88.71% ± 0.39%
	HGEN-Motif	932.12 ± 7.58	53.1 ± 4.66	4.66e–3 ± 0.0	2.23 ± 0.04	−0.21 ± 0.03	1.29e–2 ± 3.13e–3	5.32% ± 0.13%	87.43% ± 0.52%
	Real	1,074	1	4.43e–4	2.51	−0.235	N/A	N/A	N/A
DBLP	NetGAN	10,353 ± 72.71	0.0 ± 0.0	0.0 ± 0.0	3.308 ± 0.41	−0.059 ± 0.03	5.03e–1 ± 2.1e–2	5.48% ± 0.32%	72.51% ± 0.32%
	VGAE	3,771 ± 236.29	1214.69 ± 452.61	0.271 ± 0.06	1.579 ± 0.07	−0.44 ± 0.11	8.71e–2 ± 1.77e–3	≈0%	17.26% ± 0.41%
	HGEN	5,163 ± 21.41	1068 ± 12.83	0.018 ± 0.001	1.793 ± 0.21	−0.157 ± 0.03	5.82e–3 ± 1.67e–4	1.55% ± 0.09%	66.59% ± 0.17%
	HGEN-Motif	5,624 ± 78.32	788 ± 65.32	0.012 ± 0.001	1.705 ± 0.14	−0.157 ± 0.03	3.57e–3 ± 2.55e–4	1.26% ± 0.1%	58.89% ± 1.79%
	Real	5,513	0.0	0.0	1.855	−0.201	N/A	N/A	N/A

Baselines.

Since no baseline is available for the novel task of heterogeneous graph generation, we carefully adapt four state-of-the-art graph generation methods: NetGAN [22], GraphVAE [23], VGAE [25], and GraphRNN [2]. We utilize node type information as node features of the input graph in GraphVAE and VGAE. In addition, we modify NetGAN and GraphRNN to make them available to generate node types. HGEN refers to the model that does not generate network motifs for the proposed method. We further compare HGEN-Motif, which generates network motifs along with meta-paths.

Evaluation Metrics.

The evaluation of heterogeneous graph generation can be divided into three aspects. 1) Graph Statistical Properties: we focus on six typical statistics as widely used in [22, 31, 36] for measuring the structural similarity, including LCC (the size of the largest connected component), TC (Triangle count), Clustering Coef. (clustering coefficient); Powerlaw Coef. (power-law distribution of the node degree distribution), Assortativity, and Degree Distribution Dist. (Node degree distribution Maximum Mean Discrepancy distance). 2) Graph Novelty and Uniqueness. Ideally, we would want the generated graphs to be diverse and similar, but not identical. To quantify this aspect, we check the uniqueness between the generated graphs by calculating their edit distances. Specifically, we align the node order between the test graph and the generated graph, and calculate the EO Rate (edge overlapping rate) between the generated graphs and the testing graphs for measuring the novelty of the generated graphs. A higher EO Rate indicates the generation method tends to generate more similar graphs than other approaches. The uniqueness is utilized to capture the diversity of generated graphs. To calculate the uniqueness of a generated graph, we let each model to generate 100 graphs, and the generated graphs that are subgraph isomorphic to some other generated graphs are first removed. The percentage of graphs remaining after this operation is defined as uniqueness. For example, if the model generates 100 graphs, all of which are identical, the uniqueness is 1/100 = 1%. 3) Meta-path Ratio Properties: We measure the preservation of meta-path distribution in two metrics. Firstly, we measure the meta-path length ratio preservation. Secondly, under different meta-path lengths, we also measure the distribution of the frequent meta-path patterns.

5.3. Quantitative Analysis

Preservation of Graph Statistical Properties.

We evaluate the performance of HGEN and its extended model HGEN-Motif against all the baselines on the standard graph statistics, and the results are shown in Table 3. Overall, HGEN and HGEN-Motif achieve competitive performance with very few exceptions on all metrics over synthetic and real-world datasets. We report several observations from the table: 1) Node-level similarity: HGEN-based models are the dominant performer in most node-level metrics. Although there are no significant differences in both Assortativity and Power-law Coef. among all the algorithms, HGEN rank top with very few exceptions in the node degree distribution distance with at least 40% improvement, which indicates that HGEN can effectively capture the degree distribution of all types of nodes through jointly learning both meta-path and random walk distribution. 2) Graph level similarity: HGEN-based models still exceed other baselines by effectively preserving the community distribution. Specifically, for all the datasets with rich local community information (e.g., PubMed and synthetic datasets), HGEN-based models can utilize the heterogeneous node embedding for preserving the higher-order structural information in the generated heterogeneous walks, which leads to better performance in metrics like LCC, TC, and Clustering Coef.. However, in heterogeneous graphs with rare high-order structures, the performance of HGEN-based models are comparatively less impressive. 3) As shown in Table 3, the random-walk based method HGEN and NetGAN can generally achieve stable performance than one-shot based (e.g., VGAE and GraphVAE) and sequential-based (GraphRNN) generative models across all datasets. The reason is that random walk based methods learn the overall graph distribution by learning the distribution of its discrete random walks, which is not sensitive to various graph characteristics. 4) Table 3 also shows that VGAE cannot produce realistic graphs even though it achieves the best performance in some metrics, which is expected since the primary purpose of VGAE is learning node embeddings but not generating entire graphs. In addition, as the size of the graph increases, GraphRNN also fails to generate realistic graphs because of the weak scalability of auto-regressive models.

Graph Novelty and Uniqueness.

The results of graph novelty and uniqueness are reported in the right two columns in Table 3. Specifically, HGEN achieves a generally lower EO rate across all datasets, indicating that HGEN does not purely memorize the seen heterogeneous walks in the training data. In contrast, GraphRNN has a higher EO rate, indicating GraphRNN regenerates graphs it saw during training. In addition, VGAE achieves the lowest EO rate since it fails to generate realistic heterogeneous graphs. For Uniqueness, HGEN also exceeds other one-shot and sequential-based algorithms by an evident margin, demonstrating the generated graphs’ diversity.

Preservation of Graph Semantic Properties

To further demonstrate the performance of HGEN, we evaluate the performance of meta-path distribution preservation with other baselines. Specifically, we measure the meta-path distribution from two aspects: 1) the overall meta-path length ratio preservation in generated graphs and 2) frequent meta-path patterns under each length. In general, all the methods can approximately maintain the meta-path length ratio except for VGAE. However, HGEN can constantly achieve a better performance as shown in Figure 7a and 7b. 2) As shown in Figure 7c – 7e and 7f – 7h, HGEN can outperform other methods by at least 10% in preserving the ratio of specific meta-path patterns under each length, which is expected since HGEN is able to learn and maintain the meta-path distribution from the observed graphs while others cannot.

Fig. 7:

The meta-path distribution comparison. 7a and 7b are the generated meta-path length distribution for Syn_500 dataset and PubMed dataset, respectively. 7c - 7e and 7f - 7h are frequent meta-path patterns distribution with length 1, 2, 3 for Syn_500 dataset and PubMed dataset, respectively.

5.4. Link Prediction

Link prediction is commonly used as an evaluation to predict the existence of unobserved links (i.e., edges) in a given observed graph, and we use it to evaluate the generalization power of HGEN and other approaches. We randomly mask out 40% of the edges as a testing set and report the performance with two commonly used metrics: area under the ROC curve (AUC) and F1 score (F1). We conducted the experiments with other approaches on all datasets; note that the Syn-Multi dataset contains multiple edge relations while other approaches cannot handle the multi-typed edge generation job. We, therefore, only compare HGEN variants.

The results are reported in Table 4. Although there is no overall dominant method, HGEN-based methods still achieve comparably more impressive performance. With the effort to preserve local semantic distribution and higher-order structural information, HGEN-based models can leverage observed heterogeneous information to complement the rest. In addition to the normal link prediction, HGEN-based models can still perform well in recovering the multi-edge type information, proving HGEN can characterize different meta-path distributions in the observed heterogeneous graph.

Table 4:

Link Prediction Performance (in %). We randomly sampled 60% edges as a training graph and the rest of the edges as testing. For Syn_Multi dataset, since no existing methods are capable of generating different meta-relations between edges, we only compare HGEN with its variant HGEN-Motif.

Method	NetGAN		GraphRNN		GraphVAE		VGAE		HGEN		HGEN-Motif
	F-1	AUC	F-1	AUC	F-1	AUC	F-1	AUC	F-1	AUC	F-1	AUC
Syn_100	73.24	88.93	79.67	89.67	54.32	78.63	66.78	87.89	78.93	92.63	81.52	93.11
Syn_200	76.29	81.63	82.54	93.41	58.67	77.63	59.63	77.89	77.92	89.68	79.25	92.66
Syn_500	81.32	89.86	80.54	92.67	77.85	84.63	61.76	83.81	73.69	89.63	84.32	93.42
PubMed	68.54	77.63	71.32	78.77	56.53	75.45	54.32	76.42	72.89	79.96	71.32	79.63
IMDB	64.96	77.82	52.42	71.63	62.53	77.64	67.21	81.53	70.83	80.82	68.32	82.78
DBLP	70.54	81.32	64.32	84.31	52.53	59.63	62.53	82.81	72.57	89.92	73.58	74.51
Syn_Multi	–	–	–	–	–	–	–	–	69.54	87.63	72.45	80.32

5.5. Ablation Study

We further conduct ablation studies on the PubMed dataset to evaluate the effect of different components in HGEN, and the results are exhibited in Table 5. The ablative experiments are conducted based on each of the essential components in our architecture. Specifically, we select a single large heterogeneous walk length - 8 to replace the heterogeneous walk length 1, 2, and 3 in our model, and the resulting model is called HGEN-S. We also independently remove the heterogeneous node embedding to let the generator uniformly sample the next node, and the resulting model is named HGEN-E. Moreover, we replace the heterogeneous graph assembler with a probability-based graph assembler, namely HGEN-A. Lastly, we add another evaluation metric - OC (Orbit Count) to quantify how HGEN and HGEN-Motif perform when preserving higher-order structures.

Table 5:

Ablation Study in PubMed Dataset

	HGEN-S	HGEN-E	HGEN-A	HGEN	HGEN-Motif	Real
LCC	1563.76	824.14	819.32	825.6	897.3	948
TC	1453.23	784.34	863.53	1569.3	1972.3	2114
OC	512.38	379.63	432.67	453.28	509.45	576
Clustering Coef.	0.026	0.015	0.016	0.034	0.051	0.068
Power Law Coef.	1.649	1.652	1.621	1.634	1.505	1.75
Assortativity	−0.09	−0.132	−0.131	−0.143	−0.162	−0.208
Node Degree Dist.	0.0354	0.0388	0.0515	0.0392	0.0521	N/A

As shown in Table 5, all the ablative models achieve similar results in node-level metrics like Powerlaw Coef., Assortativity, which is because HGEN can well capture this node-level information through learning the heterogeneous walk distribution. Other than that, we observe: 1) HGEN-S can construct a larger sub-graph since the length of the heterogeneous walk is largely greater than HGEN, but the large subgraph doe not makes any improvements in terms of capturing the heterogeneous structural information. The reason is there are rarely long meta-path in the heterogeneous graph since longer meta-paths are highly redundant because of the shared sub-parts [11]. We instead choose 1, 2, and 3 as our meta-path lengths to make the whole generation more flexible. 2) removing the heterogeneous node embedding would make HGEN-E hard to capture the local graph structure since HGEN relies on the encoded neighborhood information to make the node sampling aware of the local structure. 3) as shown in the node degree distribution evaluation, replacing the heterogeneous graph assembler with a probabilistic graph assembler would cause HGEN-A hard to capture the latent heterogeneous node distribution because it uniformly samples edges from the generated walks and completely neglects the generated meta-path information. However, HGEN takes meta-paths as a basic unit to sample edges so that it can effectively preserve the overall distribution of meta-paths as proved in Theorem 1. Therefore, the node degree distribution under each type can be well preserved. Finally, HGEN-Motif performs better than HGEN in generating the most similar triangle and orbit counts with the observed graph, which also justifies the choice of adding motif as the based generation unit.

5.6. Running Time Comparison

Figure 8 shows the results of our running time experiments. The running times on both synthetic and real-world datasets, including both training and inference time, are shown with respect to the growth of the number of nodes in both synthetic and real-world datasets. All running times are in the log−10 scale. As shown in both figures, random-walk-based generative models (HGEN and NetGAN) have a constant running time growth in terms of the number of nodes, which is especially important when dealing with large graphs. Even though VGAE is much faster in running time, it is indeed a representation learning framework based on GCN and lacks the ability to generate realistic heterogeneous graphs, and the results are also reflected in Table 3. Both GraphRNN and GraphVAE fail to compare with HGEN in model scalability because their designs require at least $O\left(|𝒱{|}^2\right)$ to process the transformed node sequence and adjacency matrix.

Fig. 8:

Running time comparison of different models in both synthetic and real-world datasets. It is clear that GraphVAE is not scalable in generating graphs with more than 200 nodes. GraphRNN also fails in generating large graphs (with more than 10, 000 nodes). The proposed HGEN exhibits a linear running time growth in terms of the growth of graph size.

5.7. Graph Visualization

Since it is nearly impossible to judge whether a graph is realistic only by statistics, we visualize the generated graph to further demonstrate the performance of HGEN (Figure 9). Visually, HGEN looks the most similar, while both GraphVAE and VGAE is the most dissimilar. This result is consistent with the quantitative results obtained in Table 3. For one-shot based generative models, GraphVAE and VGAE, they fail to capture the structural similarity of the observed heterogeneous graph. For the sequential-based and random walk based graph generative methods, GraphRNN and NetGAN can successfully mimic the structure similarity but fail to preserve the global heterogeneous graph properties (e.g., overall meta-path ratio).

Fig. 9:

9a - 9f are the generated graph of the Syn_100 dataset, 9g – 9l are the generated graph of the Syn_200 dataset, 9m - 9q are the generated graph of the Syn_500 dataset, and 9r - 9v are the generated graphs of the PubMed dataset.

6. Conclusion

This paper focuses on a new problem: heterogeneous graph generation. To achieve this, we propose a novel framework - HGEN for the heterogeneous graph generation. Specifically, the proposed method consists of a novel heterogeneous walk/motif generator that can hierarchically generate meta-paths and a heterogeneous graph assembler that can construct new graphs by sampling from the generated heterogeneous walks in a stratified manner. As the extension of the meta-path-based HGEN, this paper proposes a novel module HGEN-Motif that considers the network motif as one of the basic generation units in order to better capture the higher-order structural distribution. We further unified the training framework to enable the generator to generate various heterogeneous instances to meet different statistics of the observed heterogeneous graph. Compared to existing deep graph generation methods, HGEN is tailored for heterogeneous graph generation and can provide more insights into heterogeneous graph mining studies. It is evaluated from the experiments that existing deep graph generation methods cannot well preserve the local semantic, higher-order structural, and global distribution of an observed heterogeneous graph, and thus cannot handle the unique job of heterogeneous graph generation. As the first-of-its-kind heterogeneous graph generation method, HGEN can not only provide benchmarks for the many heterogeneous graph-related studies, but it can also enrich our understanding of the implicit properties of heterogeneous graphs.

Fig. 5:

The illustration of the extended heterogeneous walk/motif generation. We separate the generation unit for meta-paths and heterogeneous network motifs, and we leverage the sampling-based method to choose each unit.