Network-based hierarchical heterogeneity analysis and applications to cancer omics data

Abstract

In the study of complex diseases, heterogeneity analysis has been routinely conducted. A series of recent studies have suggested that network (graph)-based heterogeneity analysis can take a system perspective and be more informative than that based on simpler statistics such as mean and variance. In this article, we conduct Gaussian graphical model (GGM)-based heterogeneity analysis. Significantly advancing from the existing literature, we consider the scenario, where measurements can be decomposed into two parts, with the first and second parts for a rough grouping and a refined subgrouping, respectively. Additionally, the groups and subgroups have a nested structure, which enhances interpretability. A penalization approach is developed for simultaneous sparse estimation, grouping and subgrouping, and achieving the hierarchical structure. Its theoretical properties are rigorously established, and an effective computational algorithm is developed. Simulation demonstrates its competitive empirical performance. The analysis of data from the Pan-Cancer Analysis of Whole Genomes (PCAWG) further demonstrates its practical utility and leads to sensible findings.

1. Introduction

Heterogeneity is a hallmark of many complex diseases. Heterogeneity analysis can help better understand disease biology, make more accurate prediction of disease paths, and design customized treatment strategies. It can be broadly classified as unsupervised and supervised heterogeneity analysis, both of which have been extensively conducted. In this study, we conduct unsupervised heterogeneity analysis, under which a set of variables has different distributional properties in different sample groups. Early heterogeneity analyses have been mostly based on low-dimensional variables, such as demographic and clinical measurements. More recent analyses have been based on high-dimensional variables, such as those from sequencing and imaging studies, and are notably more challenging. Analysis can be based on “simpler” statistics such as mean and variance. In a series of recent studies, network (graph)-based analysis has been conducted [3, 6, 12]. Such analysis can incorporate mean and variance (and possibly other statistics), define sample differences based on the interconnections among variables, and provide a deeper/more comprehensive view.

In network(graph)-based heterogeneity analysis, the most commonly adopted technique is the Gaussian graphical model (GGM). In some studies [5, 7], the GGM-based finite mixture modeling is conducted, and the number of sample groups is pre-fixed or determined using criteria such as Bayesian information criterion (BIC). To more “automatedly” determine the number of groups and provide a more solid statistical ground, studies such as [11, 18] adopt the penalized fusion technique.

In the aforementioned and some other studies, a single level of grouping is conducted, and samples are classified into mutually exclusive groups only once. In “classic” clustering analysis, hierarchical clustering has been popular, under which sample groups can be further divided into subgroups that are nested within the groups. This process can be further repeated. Compared with partitional clustering, hierarchical clustering, by looking at sample grouping at multiple levels/resolutions, can sometimes be more informative. The scheme of the proposed analysis is shown in Figure 1. Motivated by the success of GGM-based heterogeneity analysis and the lack of hierarchical structure in such analysis, our goal is to identify sample groups and subgroups, with subgroups nested in groups, and different groups/subgroups are defined by their unique network structures. To simplify notation, we consider two levels in this article and note that the proposed analysis can be potentially extended to more levels.

Figure 1.

(Color online) Scheme of analysis. Left: subjects are divided into two groups. Group-A is further divided into three subgroups (A1, A2, and A3); the two subgroups of Group-B are omitted to simplify the presentation. Middle: the group-level networks, the subgroup-level networks, and the networks combining the two levels. Right: the observed networks without distinguishing the two levels

Some intuition can be inferred from Figure 1. For the samples in each subgroup, their overall network structure (“Combined” in Figure 1) is jointly determined by the network structures of the subgroup and the corresponding upper-level group. The “intertwinement” of networks can be complex, meaning that it is possible that the observed networks (“Observed” in Figure 1) can be dense even though the corresponding networks at different levels are sparse. There is no definitive evidence in published data analyses regarding the relative sparsity of networks at different levels. In genetic data analysis, the group-level analysis may correspond to gene pathways, and the subgroup-level analysis may correspond to more refined functional units. It has been indicated that the subgroup-level networks may be sparser than their group-level counterparts. However, this is not necessarily true for other data scenarios.

To achieve two (or more) levels of grouping, one possibility is to analyze the same measurements with different resolutions, as in, for example, classic hierarchical clustering. Our preliminary exploration suggests that, with GGM, it is challenging to conduct this type of analysis, simultaneously achieve multiple levels of grouping, and establish consistency properties. Another possibility is to use different data/information for different levels. For example, pathological imaging data are used for grouping, and omics data are used for subgrouping¹⁾. This type of analysis demands two or more types of data. Moreover, [9] considers a direct decomposition of precision matrices into common parts and subgroup-specific parts for different subgroups, requiring that the network of a group be the common structure of the subgroup networks it contains. In this study, we take a different strategy that complements the existing ones. In particular, it is assumed that the observed measurements can be decomposed into two parts, with the first providing rough information and forming grouping and the second providing more subtle information and forming subgrouping. Intuitively, when we take a rough look at data, a rough grouping can be obtained. Subsequently, when we take a closer look at data, more subtle information, which is not reflected in rough grouping, is obtained and leads to more refined subgrouping. As subgrouping is meant to improve over grouping as opposed to voiding it, a nested structure is desirable. Similar data decomposition strategies have been considered in [17,22,25] under different contexts, and note that they are all based on single-level grouping with known subgroups.

Overall, this study aims to develop a novel alternative heterogeneity analysis tool based on GGM. Similar to the other GGM-based heterogeneity analysis studies, it can be more informative than those based on simpler statistics. Advancing from the existing GGM-based heterogeneity analyses such as [3, 14, 15], it conducts a multi-level analysis, with lower-level subgroups nested in higher-level groups. As discussed above, our strategy of utilizing information for multi-level grouping differs from and complements some of the existing literature. Also, it is noted that, although the concept of hierarchical clustering is classic, conducting such analysis in a single step (as opposed to multiple steps) based on GGM is highly nontrivial. This study also differs from and complements, for example, the supervised regression-based hierarchical heterogeneity analysis [8]. The rigorous theoretical and computational developments can also broadly shed insights on GGM, hierarchical clustering, high-dimensional penalized fusion, and other areas. Last but not least, as demonstrated in data analysis, this study can deliver a practically useful tool and advance cancer omics and other research.

2. Model

2.1. Data and modeling framework

Consider the scenario described above, where measurements are available on $p$ variables for $n$ independent samples. Assume that the samples can be divided into a total of $K_0$ subgroups as well as a total of $K_1$ groups. The subgroups are nested within the groups, i.e., there is a hierarchical structure. There is no constraint on the number of subgroups within each group.

Let ${\mathit{X}}_{k,i}$ be the vector of measurements for subject $i$ in the $k$-th subgroup. Assume ${\mathit{X}}_{k,i}={\mathit{Y}}_{m(k),i}+{\mathit{Z}}_{k,i}$, where ${\mathit{Z}}_{k,i}$ is the random vector corresponding to the $k$-th subgroup-specific effect, and ${\mathit{Y}}_{m(k),i}$ is the random vector corresponding to the $m(k)$-th group-specific effect, which is identically distributed across the subgroups with the same $m(k)$ value. Assume that ${\mathit{Y}}_{m(k),i}$ and ${\mathit{Z}}_{k,i}(i=1,\dots ,n;k=1,\dots ,K_0;m(k)=1,\dots ,K_1<K_0$) are $p$-dimensional Gaussian random vectors with covariance matrices ${𝚺}_{Y_{m(k)}}$ and ${𝚺}_{Z_k}$, respectively. Let ${𝚽}_k={\left({𝚺}_{Z_k}\right)}^{-1}$ and ${𝚯}_k={\left({𝚺}_{Y_{m(k)}}\right)}^{-1}$, with ${𝚯}_{k_1}={𝚯}_{k_2}$ if $m\left(k_1\right)=m\left(k_2\right)$ (i.e., $k_1$-th and $k_2$-th subgroups belong to the same group). Additionally, assume that ${\mathit{Y}}_{m(k),i}$ and ${\mathit{Z}}_{k,i}$ are independent.

The goal of the analysis is to identify the hierarchical structure and estimate the group- and subgroup-specific precision matrices ${𝚯}_k’\text{s}$ and ${𝚽}_k’\text{s}$ based on the observed ${\mathit{x}}_i’\text{s}$. Here, we consider the challenging scenario, where $K_0$ and $K_1$ are unknown. ${\mathit{y}}_i$ and ${\mathit{z}}_i$, the $i$-th realization of $\mathit{Y}$ and $\mathit{Z}$, are the latent variables. Drawn from independent samples, ${\mathit{y}}_i’\text{s}$ and ${\mathit{z}}_i’\text{s}$ are independent for different $i’\text{s}$. Under the assumed setting, the $k$-th subgroup has distribution $𝒩\left({\mathit{\mu }}_k,{\left({𝚯}_k{\left({𝚯}_k+{𝚽}_k\right)}^{-1}{𝚽}_k\right)}^{-1}\right)$ and density $f_k\left(\mathit{x};{\mathit{\mu }}_k,{𝚯}_k^{-1},{𝚽}_k^{-1}\right)$, where the mean vector and covariance matrices are unknown. Note that this density is determined by both the subgroup- and group-level parameters. As each subgroup corresponds to one specific group, once the subgroup index is set, both the corresponding subgroup and group parameters are determined. Then, ${\mathit{x}}_i$, the $i$-th realization of $\mathit{X}$, has density $f(\mathit{x})=\sum _{k=1}^{K_0}{\pi }_k^{*}{f}_k\left(\mathit{x};{\mathit{\mu }}_k,{𝚯}_k^{-1},{𝚽}_k^{-1}\right)$, where the mixture probabilities ${\pi }_k^{*}’\text{s}$ are unknown. Here, to accommodate heterogeneity, a GGM-based mixture modeling is adopted. It is assumed that an observed measurement vector can be decomposed into a higher group-level component and a lower subgroup-level component. For both components, the Gaussian distributions are assumed. The numbers of subgroups and groups are not assumed to be known, making the proposed model more flexible. The hierarchical grouping structure includes the single-level heterogeneity analysis as a special case.

2.2. Hierarchical penalty estimation

To estimate the model parameters and identify the hierarchical grouping-subgrouping structure, we consider the penalized objective function

$$ℒ(𝛀,\mathit{\pi }\mid \mathbb{X})=\frac{1}{n}\sum _{i=1}^n\text{log}\left(\sum _{k=1}^K{\pi }_k{f}_k\left({\mathit{x}}_i;{\mathit{\mu }}_k,{𝚯}_k^{-1},{𝚽}_k^{-1}\right)\right)-𝒫(𝛀),$$ (2.1)

where $\mathbb{X}$ denotes the collection of the observed data, $K$ is a user-set “upper bound” of the number of subgroups, $𝛀={\left({𝛀}_1^{\top },\dots ,{𝛀}_K^{\top }\right)}^{\top },{𝛀}_k=\text{vec}\left({\mathit{\mu }}_k,{𝚯}_k,{𝚽}_k\right)=\left({\mu }_{k1},\dots ,{\mu }_{kp},{\theta }_{k11},\dots ,{\theta }_{kp1},\dots ,{\theta }_{k1p},\dots ,{\theta }_{kpp}\right.,\left.{\varphi }_{k11},\dots ,{\varphi }_{kp1},\dots ,{\varphi }_{k1p},\dots ,{\varphi }_{kpp}\right)\in {\mathbb{R}}^{2p^2+p},{𝚽}_k$ is the precision matrix of the $k$-th subgroup with the $ij$-th entry ${\varphi }_{kij}$, and ${𝚯}_k$ is the precision matrix of the $k$-th group with the $ij$-th entry ${\theta }_{kij}$. In addition, $\mathit{\pi }={\left({\pi }_1,\dots ,{\pi }_K\right)}^{\top }$. Furthermore, define ${𝛀}_k^u=\text{vec}\left({𝚯}_k\right)$ and ${𝛀}_k^l=\text{vec}\left({\mathit{\mu }}_k,{𝚽}_k\right)$. For the penalty we have,

$$𝒫(𝛀)=\sum _{k=1}^K\left(\sum _{j=1}^{p}p\left(\left|{\mu }_{kj}\right|,{\lambda }_1\right)\right)+\sum _{k=1}^K\left(\sum _{i\ne j}p\left(\left|{\theta }_{kij}\right|,{\lambda }_2\right)+\sum _{i\ne j}p\left(\left|{\varphi }_{kij}\right|,{\lambda }_2\right)\right)+\sum _{k<k^{\prime }}p\left({\left({‖{𝛀}_k^u-{𝛀}_{k^{\prime }}^u‖}_2^2+{‖{𝛀}_k^l-{𝛀}_{k^{\prime }}^l‖}_2^2\right)}^{1/2},{\lambda }_3\right)+\sum _{k<k^{\prime }}p\left({‖{𝛀}_k^u-{𝛀}_{k^{\prime }}^u‖}_2,{\lambda }_4\right),$$ (2.2)

where $p(\cdot ,\lambda )$ is a concave penalty function with a tuning parameter $\lambda (\lambda >0)$, and ${\lambda }_1,{\lambda }_2,{\lambda }_3$, and ${\lambda }_4$ are tuning parameters. In our numerical study, we adopt the minimax concave penalty (MCP) [26] and note that some other penalties such as smoothly clipped absolute deviation (SCAD) are also applicable. In most if not all scenarios, although it is hard to know $K_0$ a priori, its “upper bound” $K$ can be easily specified, and $K$ can be set as a relatively large number, as shown in our numerical analysis. Consider the estimate

$$(\widehat{𝛀},\widehat{\mathit{\pi }})=\underset{𝛀,\mathit{\pi }}{\text{argmax}}ℒ(𝛀,\mathit{\pi }\mid \mathbb{X}).$$

Let $\left\{{\widehat{𝚼}}_1,\dots ,{\widehat{𝚼}}_{{\widehat{K}}_0}\right\}$ be the distinct values of $\widehat{𝛀}$ and $\left\{{\widehat{𝚪}}_1,\dots ,{\widehat{𝚪}}_{{\widehat{K}}_1}\right\}$ be the distinct values of $\widehat{𝚯}$.

$${\left\{{\widehat{𝒯}}_l\right\}}_{l=1,\dots ,{\widehat{K}}_0}={\left\{k:{\widehat{𝛀}}_k\equiv {\widehat{𝚼}}_l,k=1,\dots ,K\right\}}_{l=1,\dots ,{\widehat{K}}_0}$$

and

$${\left\{{\widehat{ℐ}}_l\right\}}_{l=1,\dots ,{\widehat{K}}_1}={\left\{k:{\widehat{\Theta }}_k\equiv {\widehat{𝚪}}_l,k=1,\dots ,K\right\}}_{l=1,\dots ,{\widehat{K}}_1}$$

both constitute partitions of $\{1,\dots ,K\}$ which is a more relaxed division, corresponding to the last two penalty terms shrinking to zero (or less than a given threshold).

Then, ${\widehat{K}}_1$ groups and ${\widehat{K}}_0$ subgroups can be identified, and their estimated precision matrices are obtained. The mixture probabilities of the subgroups and groups, respectively, ${\widehat{\mathit{\pi }}}^l={\left({\widehat{\pi }}_1^l,\dots ,{\widehat{\pi }}_{{\widehat{K}}_0}^l\right)}^{\top }$ and ${\widehat{\mathit{\pi }}}^u={\left({\widehat{\pi }}_1^u,\dots ,{\widehat{\pi }}_{{\widehat{K}}_1}^u\right)}^{\top }$, can be extracted from $\widehat{\mathit{\pi }}$. To better characterize the nested group-subgroup relationships, we conduct a mutually exclusive partition of $\left\{1,\dots ,{\widehat{K}}_0\right\}:\left\{{\widehat{\mathscr{H}}}_1,\dots ,{\widehat{\mathscr{H}}}_{{\widehat{K}}_1}\right\}$, satisfying

$${\widehat{ℐ}}_{l_2}=\bigcup _{l_1\in {\widehat{\mathscr{H}}}_{l_2}}{\widehat{𝒯}}_{l_1},1<l_1⩽{\widehat{K}}_0,1⩽l_2⩽{\widehat{K}}_1.$$

This indicates that the subgroups are indexed under (nested within) the groups. Note that a smaller ${\widehat{K}}_1$ does not necessarily mean a smaller ${\widehat{K}}_0$.

Rationale.

The proposed objective function has an intuitive formulation. The first term is the log-likelihood function under the Gaussian mixture model. The first two terms in the penalty function lead to sparsity and have been commonly adopted in the literature [4,16]. It is noted that under heterogeneity, the means cannot be eliminated via normalization because it is impossible to achieve equal means using centralization. When sparsity in the means is not of interest, the first penalty term can be removed, which can simplify computation.

The most significant advancement is the last two penalty terms. For heterogeneity analysis, we adopt the penalized fusion strategy, which has been popular in recent studies [19]. Its most significant advantage is that it “transforms” the determination of the number of groups (which is usually challenging) to a tuning parameter selection problem, which can be more “automated”. Additionally, the theoretical properties can be easier to establish. Here, it is noted that most of the existing penalized fusion-based heterogeneity analysis has only one level, without considering the hierarchical structure. In data analysis, although the numbers of groups and subgroups are unknown, it is usually not difficult to specify an “upper bound”. Once $K$ subgroups are identified, the penalized fusion encourages smaller subgroups/groups to be merged into bigger ones.

To respect the hierarchy, we adopt a strategy that shares some similar spirit with the hierarchically structured sparsity penalization [1]. With the proposed penalty and the initial $K$ subgroups, the resulting estimates have the following possibilities:

1. ${𝛀}_k^u={𝛀}_{k^{\prime }}^u$ and ${𝛀}_k^l={𝛀}_{k^{\prime }}^l$: the initial subgroups $k$ and $k^{\prime }$ are merged into one group and one subgroup, and the density functions for the two subgroups are exactly the same;

2. ${𝛀}_k^u\ne {𝛀}_{k^{\prime }}^u$ and ${𝛀}_k^l\ne {𝛀}_{k^{\prime }}^l$: the initial subgroups $k$ and $k^{\prime }$ belong to different groups and different subgroups;

3. ${𝛀}_k^u={𝛀}_{k^{\prime }}^u$ and ${𝛀}_k^l\ne {𝛀}_{k^{\prime }}^l$: the initial subgroups $k$ and $k^{\prime }$ belong to different subgroups but the same group.

It is noted that, with the “all in or all out” property of group penalization, the case with ${𝛀}_k^u\ne {𝛀}_{k^{\prime }}^u$ and ${𝛀}_k^l={𝛀}_{k^{\prime }}^l$ cannot happen. As such, the hierarchy is respected.

3. Theoretical properties

Denote the true values of the parameters by

$${𝛀}^{*}={\left({𝛀}_1^{*\top },\dots ,{𝛀}_K^{*\top }\right)}^{\top },{𝛀}_k^{*}=\text{vec}\left({\mathit{\mu }}_k^{*},{𝚯}_k^{*},{𝚽}_k^{*}\right)\in {\mathbb{R}}^{2p^2+p},1⩽k⩽K_0.$$

Let ${𝚼}^{*}=\left\{{𝚼}_1^{*},\dots ,{𝚼}_{K_0}^{*}\right\}$ be the distinct values of ${𝛀}^{*}$ corresponding to the subgroups. Also let ${𝚯}^{*}={\left(\text{vec}\left({𝚯}_1^{*\top }\right),\dots ,\text{vec}\left({𝚯}_K^{*\top }\right)\right)}^{\top }$ and $\text{vec}\left({𝚯}_k^{*\top }\right)\in {\mathbb{R}}^{p^2}$. Let ${𝚪}^{*}=\left\{{𝚪}_1^{*},\dots ,{𝚪}_{K_1}^{*}\right\}$ be the distinct values of ${𝚯}^{*}$ corresponding to the groups. Define the minimum differences of the common parameters between two subgroups and two groups as $b_1={\text{min}}_{1⩽l_1\ne l_2⩽K_0}{‖{𝚼}_{l_1}^{*}-{𝚼}_{l_2}^{*}‖}_2$ and $b_2={\text{min}}_{1⩽l_1\ne l_2⩽K_1}{‖{𝚪}_{l_1}^{*}-{𝚪}_{l_2}^{*}‖}_2$, respectively. For $l_1=1,\dots ,K_0$ and $l_2=1,\dots ,K_1$, define ${𝒮}_{l_1}=\left\{(i,j):{\varphi }_{l_{1}ij}^{*}\ne 0,1⩽i\ne j⩽p\right\},{𝒮}_{l_2}^{\prime }=\left\{(i,j):{\theta }_{l_{2}ij}^{*}\ne 0,1⩽i\ne j⩽p\right\}$, and ${𝒟}_{l_1}=\left\{j:{\mu }_{l_{1}j}^{*}\ne 0,1⩽j⩽p\right\}$. Denote by $|\cdot |$ the cardinality of a set, and define the sparsity parameters $s=\text{max}\left\{\text{max}\left\{\left|{𝒮}_l\right|,l=1,\dots ,K_0\right\},\text{max}\left\{\left|{𝒮}_l^{\prime }\right|,l\right.\right.\left.\left.=1,\dots ,K_0\right\}\right\}$ and $d=\text{max}\left\{\left|{𝒟}_l\right|,l=1,\dots ,K_0\right\}$. In the supporting information, we describe the assumed conditions, which are mild and comparable to those in the literature under simpler contexts. The following consistency properties can be established.

Theorem 3.1. Suppose that Conditions 1–5 (supporting information) hold. Assume

$$\begin{gathered} \frac{K^2{K}_0^3(s+p)\text{log}p}{n}=o(1),K_0^{1.5}\sqrt{\frac{(s+p)\text{log}p}{n}}=O\left({\lambda }_1\right)=O\left({\lambda }_2\right), \\ \text{min}\left\{{\lambda }_3,{\lambda }_4\right\}\gg K{K}_0^{1.5}\sqrt{\frac{(s+p)\text{log}p}{n}}, \end{gathered}$$

the minimum signal in the true parameters ${𝚼}^{*}$ is larger than $(a+0.5)\cdot \text{max}\left\{{\lambda }_1,{\lambda }_2\right\},b_1>(a+0.5){\lambda }_3$, and $b_2>(a+0.5){\lambda }_4$, where $a$ is the regularization parameter of the concave penalty defined in Condition 4. Then, there exists a local maximizer of (2.1) that satisfies

1. (consistency of hierarchical structure identification)

   $$\mathbb{P}\left(\widehat{K_0}=K_0,\widehat{K_1}=K_1\right)\to 1,\mathbb{P}\left({\widehat{\pi }}_{l_2}^u=\sum _{l_1\in {\widehat{\mathscr{H}}}_{l_2}}{\widehat{\pi }}_{l_1}^l,1<l_1⩽{\widehat{K}}_0,1⩽l_2⩽{\widehat{K}}_1\right)\to 1;$$
2. (estimation consistency)

   $$\sum _{k=1}^{{\widehat{K}}_0}{‖{\widehat{𝚼}}_k-{𝚼}_k^{*}‖}_2=O_p\left(K_0^{1.5}\sqrt{(d+s+p)\text{log}p/n}\right)$$

3. (sparsistency) denote the non-zero element (support) sets for parameter estimates by ${\widehat{𝒟}}_{l_1}=\left\{j:{\widehat{\mu }}_{l_{1}j}\right.\ne 0,1⩽j⩽p\},{\widehat{𝒮}}_{l_1}=\left\{(i,j):{\widehat{\varphi }}_{l_{1}ij}\ne 0,1⩽i\ne j⩽p\right\}$, and ${\widehat{𝒮}}_{l_2}^{\prime }=\left\{(i,j):{\widehat{\theta }}_{l_{2}ij}\ne 0,1⩽i\ne j⩽p\right\}$. Then, with probability tending to 1, ${\widehat{𝒟}}_{l_1}={𝒟}_{l_1}$ and ${\widehat{𝒮}}_{l_1}={𝒮}_{l_1}$ for all $l_1=1,\dots ,K_0;{\widehat{𝒮}}_{l_2}^{\prime }={𝒮}_{l_2}^{\prime }$ for all $l_2=1,\dots ,K_1$.

The proof is provided in the supporting information. In heterogeneity analysis, the label-switching (“reshuffling” of (sub)groups) problem is commonly encountered and has been well discussed in the literature. It does not affect the estimation consistency of the model parameters. This theorem establishes three much-desired consistency properties. In particular, the mixture structures at the two levels can be consistently estimated. The convergence rate $O_p(\sqrt{(d+s+p)\text{log}p/n})$ (when $K$ is bounded) is the minimax optimal rate under the Frobenius norm [2] and is similar to that in the existing literature. Additionally, the nonzero components in the means and precision matrices can be correctly identified.

We note that the present data and model settings are much more complicated and include some existing ones as special cases. As can be seen from the supporting information, the theoretical developments are not trivial. Additionally, it is noted that the heterogeneity hierarchy automatically holds by the design of the penalty.

4. Computational algorithm

We develop an effective algorithm that nests the expectation-maximization (EM) algorithm and the alternating direction method of multipliers (ADMM) algorithm. It is noted that some of the computational developments may be specific to MCP, and minor changes are needed for the other penalties. We define the latent membership matrix $\mathit{\omega }={\left({\omega }_{ik}\right)}_{n\times K_0}$, where ${\omega }_{ik}\in \{0,1\}$ is an indicator showing whether the $i$-th observation comes from the $k$-th mixing component.

First, we present the EM algorithm, which is popular with mixture models. The complete data penalized log-likelihood function is

$$\frac{1}{n}\sum _{i=1}^n\sum _{k=1}^K{\omega }_{ik}\left[\text{log}{\pi }_k+\text{log}{f}_k\left({\mathit{x}}_i;{\mathit{\mu }}_k,{𝚯}_k^{-1},{𝚽}_k^{-1}\right)\right]-𝒫(𝛀).$$ (4.1)

In the $t$-th E-step, the conditional expectation of (4.1) with respect to $\mathit{\omega }$ is

$${\mathbb{E}}_{\mathit{L}\mid \mathbb{X},{𝛀}^{(t-1)}}[ℒ(𝛀,\mathit{\pi }\mid \mathbb{X})]=\frac{1}{n}\sum _{i=1}^n\sum _{k=1}^K{L}_{ik}^{(t)}\left[\text{log}{\pi }_k+\text{log}{f}_k\left({\mathit{x}}_i;{\mathit{\mu }}_k,{𝚯}_k^{-1},{𝚽}_k^{-1}\right)\right]-𝒫(𝛀),$$ (4.2)

where $L_{ik}^{(t)}$ is the conditional expectation of ${\omega }_{ik}$ and can be computed as $L_{ik}^{(t)}=\frac{{\pi }_k^{(t-1)}{f}_k\left({\mathit{x}}_i,{𝛀}_k^{(t-1)}\right)}{\sum _{k=1}^{K_0}{\pi }_k^{(t-1)}{f}_k\left({\mathit{x}}_i,{𝛀}_k^{(t-1)}\right)}$.

The M-step aims to maximize (4.2) with respect to ($\mathit{\pi },𝛀$). For $\mathit{\pi }$, we have

$${\pi }_k^{(t)}=\frac{1}{n}\sum _{i=1}^n{L}_{ik}^{(t)}.$$ (4.3)

Maximizing (4.2) with respect to $\mathit{\mu }$ is equivalent to solving

$${\mathit{\mu }}^{(t)}=\underset{\mathit{\mu }}{\text{argmin}}\left(\frac{1}{2n}\sum _{i=1}^n\sum _{k=1}^K{L}_{ik}^{(t)}\left[{\left({\mathit{x}}_i-{\mathit{\mu }}_k\right)}^{\top }{𝚯}_k{\left({𝚯}_k+{𝚽}_k\right)}^{-1}{𝚽}_k\left({\mathit{x}}_i-{\mathit{\mu }}_k\right)\right]+𝒫(𝛀)\right),$$ (4.4)

where the local quadratic approximation is adopted, leading to an explicit solution.

Since $\mathit{Y}$ and $\mathit{Z}$ are latent (unobserved) variables, the maximum likelihood estimation method cannot be used directly. We embed another EM algorithm (inner-EM) to estimate the covariance matrices corresponding to the latent variables, and this significantly differs from the existing GGM-based heterogeneity studies. As we treat $\mathbb{Y}$ and $\mathbb{Z}$ as missing data, the complete log-likelihood function without the penalty can be rewritten as

$$l(\mathbb{Y},\mathbb{Z}\mid 𝛀)=\sum _{k=1}^K{\tilde{n}}_k\left[\text{log}\left\{\text{det}\left({𝚯}_{m(k)}\right)\right\}-\text{tr}\left({\mathit{S}}_{m(k)}^u{𝚯}_{m(k)}\right)\right]+\sum _{k=1}^K{\tilde{n}}_k\left[\text{log}\left\{\text{det}\left({𝚽}_k\right)\right\}-\text{tr}\left({\mathit{S}}_k^l{𝚽}_k\right)\right],$$ (4.5)

where ${\tilde{n}}_k=\sum _{i=1}^n{L}_{ik}^{(t)}$, and ${\mathit{S}}_{m(k)}^u$ and ${\mathit{S}}_k^l$ are the covariance matrices corresponding to ${𝚯}_k$ and ${𝚽}_k$, respectively. Each iteration of the inner-EM algorithm also consists of an E-step and an M-step. The inner-E-step calculates ${𝒬}_{\mathit{\text{in}}}(𝛀)={\mathbb{E}}_{\mathbb{Y},\mathbb{Z}\mid \mathbb{X}}l(\mathbb{Y},\mathbb{Z}\mid 𝛀)$.

The M-step maximizes the ${𝒬}_{\mathit{\text{in}}}$ function to obtain the estimated covariances. More details on the inner-EM can be found in the supporting information.

After the inner-EM iterates to convergence, we can obtain the resulting covariance matrices ${\tilde{\mathit{S}}}_{m(k)}^u$ and ${\tilde{\mathit{S}}}_k^l(k=1,\dots ,K)$. As mentioned in [22], a regularization strategy can be applied to adjust the covariance matrices and obtain improved results. For updating the precision matrices based on ${\tilde{\mathit{S}}}_{m(k)}^u$ and ${\tilde{\mathit{S}}}_k^l$, we have

$$\left({𝚯}^{(t)},{𝚽}^{(t)}\right)=\underset{𝚯,𝚽}{\text{argmax}}\left(\sum _{k=1}^K{\tilde{n}}_k\left[\text{log}\left\{\text{det}\left({𝚯}_{m(k)}\right)\right\}-\text{tr}\left({\tilde{\mathit{S}}}_{m(k)}^u{𝚯}_{m(k)}\right)\right],+\sum _{k=1}^K{\tilde{n}}_k\left[\text{log}\left\{\text{det}\left({𝚽}_k\right)\right\}-\text{tr}\left({\tilde{\mathit{S}}}_k^l{𝚽}_k\right)\right]-𝒫(𝚯,𝚽)\right).$$ (4.6)

The solution to (4.6) can be effectively obtained using the ADMM technique. Compared with [3] and some other studies, the present optimization faces more challenges with the fusion penalty. To tackle this problem, we resort to the sparse alternating minimization algorithm [23]. More details are provided in the supporting information.

Although complicated, the proposed algorithm is built on well-developed techniques, and its computational complexity has been thoroughly examined in the literature. In each iteration, all samples are involved in the E-step, and the computational complexity is expected to be $O\left(n{p}^2\right)$. For the $p$ variables in the M-step, the computational cost is expected to be $O\left(p^3\right)$ caused by matrix multiplication, inversion, and eigenvalue decomposition. Without penalization, the computational cost is expected to increase linearly with the number of mixing components. With the pairwise penalization, it is expected to be $O\left(K^2{p}^2\right)$. Convergence is expected with a moderate number of iterations. As such, we expect the overall computational complexity to be $O\left(K{p}^3+Kn{p}^2+K^2{p}^2\right)$.

The initialization is implemented using the classic hierarchical clustering method, and some other sensible clustering methods can also generate satisfactory initial values, like K-means (note that this type of Flat clustering approach may require similar subgroups to be synthesized again to initialize groups). To improve accuracy, we advise timely merging after several iterations, which is similar to the refitting step in [18], avoiding the possibility of having too few samples in a subgroup. Additionally, it is noted that a too large $K$ may lead to poor initial values (some initial subgroups may have too few samples, which may negatively affect convergence). Convergence properties of this algorithm can be established following [7]. In all of our numerical studies, convergence is satisfactorily achieved with a moderate number of iterations.

Tuning parameter selection.

We adopt a linear search and optimize an adaptive AIC-type criterion

$$-\frac{1}{n}\sum _{i=1}^n\text{log}\left(\sum _{k=1}^{{\widehat{K}}_0}{\pi }_k{f}_k\left({\mathit{x}}_i;{\widehat{𝛀}}_k\right)\right)+\sum _{k=1}^{{\widehat{K}}_0}\left(\left|{\widehat{𝒟}}_k\right|+2\left|{\widehat{𝒮}}_k\right|\right)+\sum _{k=1}^{{\widehat{K}}_1}\left(2\left|{\widehat{𝒮}}_k^{\prime }\right|\right),$$

where the degrees of freedom of the precision matrices are associated with smaller weights, following [3]. In numerical studies, we follow the published literature and fix the regularization parameter of MCP. The sparsity levels of the precision matrices at different levels may differ attributable to the sample sizes, and we adjust ${\lambda }_2$ as in [22] by multiplying $\sqrt{\left|{\mathscr{H}}_l\right|}$ for ${\Phi }_k$ if $k\in {\mathscr{H}}_l,k=1,\dots ,{\widehat{K}}_0$. In addition, the proposed method focuses more on networks than means, and so we fix ${\lambda }_1$ to be a small constant.

5. Simulation studies

For the numbers of groups and subgroups and sample sizes, we consider three scenarios: the all-balanced design with $K_0=4,K_1=2$, and sample size $n=((200,200),(200,200))$, the within-group-balanced design with $K_0=5,K_1=2$, and $n=((200,200,200),(200,200))$, and the imbalanced design with $K_0=5,K_1=2$, and $n=((250,200,150),(250,150))$. We set $p=100$. It is noted that, with the network-based analysis and heterogeneity, the number of unknown parameters is much larger than the total sample size.

The observations are generated as follows. For grouping, the memberships ${\zeta }_i’\text{s}$ are randomly sampled from {1, 2}. Then, ${\mathit{y}}_i~𝒩\left(\mu \left({\zeta }_i\right),\Sigma \left({\zeta }_i\right)\right)$. The first $2m$ components of the group means are set as 0, and the rest $p-2m$ components are set as 2 if ${\zeta }_i=1$ and 0 if ${\zeta }_i=2$. For subgrouping, if ${\zeta }_i=1$, the subgroup-membership ${\iota }_i$ is randomly sampled from {1, 2, 3} if $K_0=5$, and ${\mathit{z}}_i~𝒩\left(\mu \left({\iota }_i\right),\Sigma \left({\iota }_i\right)\right)$. The first $2m$ components of the subgroup means are set as $0\cdot 1_{2m}I\left({\iota }_i\right.=1)+2\cdot {\left(1_m^{\top },-1_m^{\top }\right)}^{\top }I\left({\iota }_i=2\right)+2\cdot {\left(-1_m^{\top },1_m^{\top }\right)}^{\top }I\left({\iota }_i=3\right)$, where $1_m$ is the length-$m$ vector of all ones. The rest $p-2m$ components are set as 0. It is noted that when $K_0=4$, there is no ${\iota }_i=1$, leading to one fewer subgroup. If ${\zeta }_i=2$, the subgroup memberships ${\iota }_i’\text{s}$ are randomly sampled from {4, 5}, and ${\mathit{z}}_i~𝒩\left(\mu \left({\iota }_i\right),\Sigma \left({\iota }_i\right)\right)$. The first $2m$ components of the subgroup means are set as $2\cdot 1_{2m}I\left({\iota }_i=4\right)-2\cdot 1_{2m}I\left({\iota }_i=5\right)$. The rest $p-2m$ components are set as $0.m=8$ in all of the simulations.

We consider three different types of networks, and each network consists of ten equal-sized disjoint subnetworks (modules) with the same properties. Among the ten subnetworks, each (sub)group has a unique subnetwork, and the rest subnetworks are shared among the (sub)groups for heterogeneity. For example, when $K_0=4$, the $i$-th subnetwork is unique for the $i$-th subgroup, and the difference between the networks of the first subgroup and the third subgroup lies in the first and third subnetworks, with the rest subnetworks being the same. For the first type of network, we consider tri-diagonal (TD) precision matrices. Here, the diagonal elements are all equal to 1, and the sub-diagonal elements are 0.3. The sub-diagonal elements of the unique subnetwork for each (sub)group are 0. For each subnetwork, we consider two popular network structures. Power-law (PL) network: the precision matrix for the first subgroup, consisting of ten power-law subnetworks, is generated through an iterative process. Starting with ten blocks, each iteration introduces two new inter-block connections. The initial matrix ${\left({\theta }_{1ij}\right)}_{q\times q}$ is generated by ${\theta }_{1ij}=1$ when $i=j,{\theta }_{1ij}=0$ when $i\ne j$ and $i\nsim j$, and ${\theta }_{1ij}=\text{Unif}([-1,-0.5]\cup [0.5,1])$ when $i\ne j$ and $i~j$, where the adjacency operator $i~j$ indicates the presence of an edge between nodes $i$ and $j$, and $i\nsim j$ indicates the absence of connection. We reinforce diagonal dominance and positive-definiteness through ${\theta }_{1jj}=\sum _{i\ne j}\left|{\theta }_{1ij}\right|+0.1$. Nearest-neighbor (NN) network: for each subnetwork of the first subgroup, we generate spatial coordinates by randomly distributing $p/10$ points within a unit square $[0,1{]}^2$, calculating all $p/10\times (p/10-1)/2$ pairwise Euclidean distances, identifying mutual $m$-nearest neighbors, and assigning non-zero entries ${\theta }_{1ij}={\theta }_{1ji}$ for connected pairs ($i,j$). The sparsity level is controlled by the neighborhood parameter $m$, and we set $m=2$. The unique network for each (sub)group is generated similarly and independently.

To make the analysis more challenging, we allow the groups and subgroups to have different types of network structures. Based on the above three network structures, we consider a total of 6 pairing combinations. It is worth noting that, although the networks are separately sparse, the observed networks after summation and inverse transformation may still be dense.

When implementing the proposed approach, we set $K=8$. To better gauge performance, we consider the following competing alternatives, all of which have been suggested as effective: (a) mixture, which is a two-step Gaussian mixture clustering approach. It assumes that the numbers of groups and subgroups (within groups) are known. It first applies the Gaussian mixture model to all samples and generates $K_0$ groups. Then the same approach is applied within each group. This approach is realized using R package “mclust”; (b) K-means, which is a two-step method that is similar to (a) but uses the silhouette method (Kmeans1) and elbow method (Kmeans2) to identify the numbers of groups and subgroups. This approach is realized using R package “NbClust”; (c) NETI2, which applies the algorithm of [22] to each group/subgroup given the oracle grouping and subgrouping memberships. The programs for this approach are obtained from [22] and its supplementary data file, which are publicly available.

Here, it is noted that, to the best of our knowledge, there is no existing approach that can directly generate the hierarchical heterogeneity structures and conduct estimation similar to the proposed one. It is possible to develop other alternative approaches (for example, by replacing K-means with another clustering technique). However, the above ones can be the most relevant. Comparatively, the proposed approach can much more easily select the numbers of groups and subgroups and estimate networks simultaneously.

To assess performance, we consider the following measures for groups and subgroups. To simplify notation, we take the subgrouping measures as an example (while the grouping measures can be defined similarly): (a) per, which is the percentage of ${\widehat{K}}_0$ equal to $K_0$; (b) mean and standard error of ${\widehat{K}}_0$; (c) subgrouping error defined by

$$\text{CE}(\widehat{\phi },\phi )≔\frac{2}{n(n-1)}\left|\left\{(i,j):I\left(\widehat{\phi }\left(x_i\right)=\widehat{\phi }\left(x_j\right)\right)\ne I\left(\phi \left(x_i\right)=\phi \left(x_j\right)\right);i<j\right\}\right|,$$

where $\widehat{\phi }$ and $\phi$ are the estimated and true subgrouping memberships, respectively; (d) mean squared error (MSE) for $\left\{\mathit{\mu }\right\}$ and $\left\{𝚽\right\}$. When ${\widehat{K}}_0=K_0$, the MSEs are defined as $\sum _{k=1}^{K_0}{‖{\widehat{\mathit{\mu }}}_k-{\mathit{\mu }}_k^{*}‖}_2/K_0$ and $\sum _{k=1}^{K_0}{‖{\widehat{𝚽}}_k-{𝚽}_k^{*}‖}_2/K_0$. When ${\widehat{K}}_0\ne K_0$, they are defined as

$$\begin{gathered} \text{MSE}(\mathit{\mu })=\frac{1}{{\widehat{K}}_0}\sum _{k=1}^{{\widehat{K}}_0}\sum _{l^{\prime }=1}^{K_0}{‖{\widehat{\mathit{\mu }}}_k-{\mathit{\mu }}_{l^{\prime }}^{*}‖}_2\cdot I\left(l^{\prime }=\underset{l}{\text{argmin}}\left\{{‖{\widehat{\mathit{\mu }}}_k-{\mathit{\mu }}_l^{*}‖}_2^2+{‖{\widehat{𝚽}}_k-{𝚽}_l^{*}‖}_F^2\right\}\right), \\ \text{MSE}(𝚽)=\frac{1}{{\widehat{K}}_0}\sum _{k=1}^{{\widehat{K}}_0}\sum _{l^{\prime }=1}^{K_0}{‖{\widehat{𝚽}}_k-{𝚽}_{l^{\prime }}^{*}‖}_F\cdot I\left(l^{\prime }=\underset{l}{\text{argmin}}\left\{{‖{\widehat{\mathit{\mu }}}_k-{\mathit{\mu }}_l^{*}‖}_2^2+{‖{\widehat{𝚽}}_k-{𝚽}_l^{*}‖}_F^2\right\}\right). \end{gathered}$$

The MSEs for $\left\{𝚯\right\}$ are similarly defined, and we use MSE1 to denote $\text{MSE}(\mathit{\mu })$ and MSE2 to denote $\text{MSE}(𝚽)+\text{MSE}(𝚯)$; (e) true and false positive rates (TPR and FPR) for the off-diagonal elements of the precision matrices. When ${\widehat{K}}_0=K_0$, they are defined as

$$\text{TPR}=\frac{1}{K_0}\sum _{k=1}^{K_0}\frac{\sum _{i<j}I\left({\varphi }_{kij}^{*}\ne 0,{\widehat{\varphi }}_{kij}\ne 0\right)}{\sum _{i<j}I\left({\varphi }_{kij}^{*}\ne 0\right)},\text{FPR}=\frac{1}{K_0}\sum _{k=1}^{K_0}\frac{\sum _{i<j}I\left({\varphi }_{kij}^{*}=0,{\widehat{\varphi }}_{kij}\ne 0\right)}{\sum _{i<j}I\left({\varphi }_{kij}^{*}=0\right)}.$$

When ${\widehat{K}}_0\ne K_0$, their definitions are revised accordingly as for the MSEs.

The proposed analysis has a more complicated objective and formulation than some of its alternatives. Hence, its computational cost is expected to be higher. However, it is still found to be affordable. For the analysis of one replicate, the proposed approach takes about 15 minutes on a laptop with standard configurations. With 100 replicates, the summary results for the unbalanced design with the nearest-neighbor networks for subgroups and the nearest-neighbor and power-law networks for groups are shown in Table 1. The other results are summarized in the supporting information. Similar observations are made across all the simulation settings.

Table 1.

Simulation results for 5 subgroups, the unbalanced design, the nearest-neighbor (NN) networks for subgroups, and the nearest-neighbor (NN)/power-law (PL) networks for groups. For each method, the first/second row corresponds to grouping/subgrouping. In each cell: mean (sd)

Network	Method	Per	${\widehat{K}}_1/{\widehat{K}}_0$	CE	MSE1	MSE2	TPR	FPR
NN-group	Mixture	1.00	2.000 (0)	0.144 (0.223)	2.785 (2.552)
NN-subgroup		1.00	5.000 (0)	0.284 (0.046)	2.988 (0.485)
	Kmeans1	0.94	2.060 (0.239)	0.006 (0.026)	1.286 (0.588)
		0	4.000 (0)	0.087 (0.010)	1.036 (0.173)
	Kmeans2	1.00	2.000 (0)	0 (0)	0.237 (0.031)
		0	4.000 (0)	0.090 (0.013)	1.046 (0.154)
	NETI2	1.00	2.000 (0)	0 (0)	0 (0)	7.775 (0.227)	0.806 (0.032)	0.174 (0.004)
		1.00	5.000 (0)	0 (0)	0 (0)	5.238 (0.104)	0.696 (0.030)	0.265 (0.005)
	Proposed	0.97	1.960 (0.200)	0.019 (0.096)	1.010 (1.410)	7.629 (0.591)	0.862 (0.033)	0.178 (0.020)
		0.91	5.080 (0.400)	0.014 (0.016)	0.808 (0.476)	5.114 (0.169)	0.775 (0.023)	0.181 (0.016)
PL-group	Mixture	1.00	2.000 (0)	0.058 (0.158)	1.794 (1.810)
NN-subgroup		1.00	5.000 (0)	0.272 (0.049)	2.818 (0.523)
	Kmeans1	0.60	2.400 (0.492)	0.046 (0.063)	2.027 (1.144)
		0	4.000 (0)	0.086 (0.012)	0.991 (0.153)
	Kmeans2	1.00	2.000 (0)	0 (0)	0.231 (0.029)
		0	4.000 (0)	0.087 (0.012)	0.974 (0.139)
	NETI2	1.00	2.000 (0)	0 (0)	0 (0)	9.270 (0.223)	0.700 (0.031)	0.178 (0.004)
		1.00	5.000 (0)	0 (0)	0 (0)	5.218 (0.114)	0.666 (0.039)	0.241 (0.007)
	Proposed	1.00	2.000 (0)	0 (0)	0.700 (0.460)	9.122 (0.198)	0.789 (0.023)	0.177 (0.012)
		0.87	5.067 (0.365)	0.009 (0.019)	0.748 (0.485)	4.915 (0.173)	0.796 (0.037)	0.180 (0.009)

The classic Gaussian mixture model has trouble identifying the correct memberships, even given the true numbers of groups and subgroups, which is reflected in its (grouping/subgrouping) classification error (CE) being significantly higher than the other methods. The two K-means approaches fail to effectively identify the correct numbers of groups and subgroups. Here, we note that determining the number of groups in unsupervised clustering has been traditionally challenging, and there is a lack of an “optimal” approach, especially under the hierarchical structure. When compared with the NETI2 approach with the true grouping for network estimation, the proposed approach, with fully data-driven (sub)groupings, has superior results. Consider, for example, Table 1, the imbalanced design, the nearest-neighbor networks for groups, and the nearest-neighbor networks for subgroups. For grouping, the proposed approach and NETI2 approach have TPR 0.862 and 0.806, FPR 0.178 and 0.174, respectively. For subgrouping, they have TPR 0.775 and 0.696, FPR 0.181 and 0.265, respectively. The proposed approach also has a small improvement in MSE2.

6. Data analysis

6.1. Analysis of ICGC/TCGA Pan-Cancer data

We analyze data from the Pan-Cancer Analysis of Whole Genomes (PCAWG) Consortium, which provides one of the most authoritative and comprehensive data sources on cancer omics. This dataset encompasses 2658 whole-cancer genomes and their matching normal tissues across 38 tumor types, amalgamated through the collaborative efforts of the International Cancer Genome Consortium (ICGC) and The Cancer Genome Atlas (TCGA) [21]. There have been multiple efforts on analyzing this dataset and its subsets [10, 13,24]. However, the perspectives taken in the existing studies are significantly different from the present one. In particular, heterogeneity analysis has been limited, and there is a lack of GGM-based analysis. Cancer heterogeneity is complicated. Samples with the same cancer can behave differently, while samples with different cancers may behave similarly. As such, it can be of interest to apply the proposed analysis to this dataset.

In principle, the proposed analysis can be conducted with a large number of genes, and this is also supported by the theoretical investigations. However, with the heterogeneity and network structures, the number of unknown parameters can be much larger than, for example, a regular regression analysis. As in the literature, we conduct a more focused analysis on a smaller number of genes. In particular, we consider the p53 signaling pathway, which was among the first to be identified as critical for multiple cancer types [20]. Genes in this pathway are essential for tumor suppression, activating DNA repair mechanisms in response to genomic damage and thus preventing the propagation of genetically compromised cells. They mediate cell cycle arrest and induce apoptosis in cells with irreparable DNA damage, serving as a critical checkpoint that can limit the development and progression of cancers by eliminating potentially cancerous cells. After standard processing, such as removing missingness, the final dataset contains 71 gene expressions measured for 1141 samples. A graphical examination suggests that, although strictly speaking the joint normality assumption may not hold, as most gene expressions have unimodal continuous distributions, the GGM technique can be reasonably applied.

When implementing the proposed approach, we set $K=8$. The initialization is conducted using hierarchical clustering. Two sample groups are identified. Furthermore, the first group contains three subgroups, with sizes 531, 73, and 43. The second group contains two subgroups, with sizes 388 and 106. Detailed grouping information is available from the authors. The network structures for the groups and subgroups are shown in Figures 2 and 3, respectively. The networks for the two groups have 580 and 473 edges, those for the subgroups of the first group have 529, 340, and 193 edges, and those for the subgroups of the second group have 501 and 333 edges. Examining the numbers of edges and network structures suggests significant differences. For example, the networks for the two groups have only 118 common edges.

Figure 2.

(Color online) Network structures for the two identified groups. (a) Group-1; (b) Group-2

Figure 3.

(Color online) Network structures for the five identified subgroups. (a) Group-1–1; (b) Group-1–2; (c) Group-1–3; (d) Group-2–1; (e) Group-2–2

A closer examination of the networks suggests more differences. For example, the properties of many individual nodes are significantly different across the networks. For example, in the networks for the two groups, genes TNFRSF10A and TNFRSF10B are important for Group-2 and Group-1, respectively, but are connected to much fewer nodes in the other networks. Compared with the other members of the TNFRSF family, TNFRSF10A and TNFRSF10B have unique sequences (p53-responsive elements) in their promoter regions that are directly regulated by p53. This means they can respond directly to p53 activation and thus play a key role in p53-mediated apoptosis signaling. TNFRSF10A is also significantly differentially expressed between the two groups of samples. This may reveal the propensity of the two groups for different apoptotic signals, reflecting the potential responsiveness of different samples to treatment. In the networks for the subgroups, PERP, which is an important gene downstream of the p53 pathway involved in the regulation of apoptosis, is an isolated node in Group-1–3 but a key hub node in the Group-1–1-1 and Group-2–1 networks, connecting with fewer nodes in Group-1–2 and Group-2–1. SERPINB5, a prognostic biomarker also known as Maspin, can be directly or indirectly up-regulated by p53, thus participating in the inhibition of tumor growth and metastasis. It is an isolated node in Group-1–2, Group-1–3, and Group-2–2, but is a key hub node in the Group-1–1 and Group-2–1 networks. Similarly, their expressions vary across the subgroups, and this differential expression may indicate the pathological characteristics and developmental stages of different types of cancers or subgroups of cancers. Information on the degrees of the identified networks is provided in Figures 1–3 in the supporting information.

A challenge faced by most if not all unsupervised heterogeneity analyses is to examine the biomedical implications of the grouping results. To this end, we compare clinical outcomes and phenotypes across the identified groups/subgroups. This approach has been commonly taken in the literature and may provide indirect support to the validity of heterogeneity analysis. We compare the survival curves, diagnosis age, and gene expressions. The results are presented in the supporting information. It is observed that the subgroups of Group-1 have significantly different survival; however, those of Group-2 do not have such a difference (see Figures 4 and 5 in the supporting information). Similar patterns are also observed for age. It may be worth mentioning that Group-1–3 has a younger age but a relatively poorer prognosis (see Figures 6 and 7 in the supporting information). The distributions of cancer types are also observed to differ significantly across the groups/subgroups (see Figures 8 and 9 in the supporting information). Additionally, for multiple genes, significant differences in expression are observed across the groups/subgroups (see Figures 10–12 in the supporting information). Such biomedically meaningful differences can suggest the practical usefulness of the proposed analysis.

6.2. Analysis with alternative methods

Data is also analyzed using the alternatives considered in the simulation. It is observed that different methods lead to significantly different results. In particular, the Gaussian mixture model leads to two groups with sizes 856 and 285, respectively, and five subgroups with sizes 719, 158, 106, 77, and 81, respectively. The Kmeans approach leads to two groups with sizes 655 and 486, respectively, and five subgroups with sizes 93, 481, 348, 103, and 116, respectively. The Kmeans + elbow approach (Kmeans2) generates fewer subgroups, while the Kmeans + silhouette approach (Kmeans1) leads to more groups and subgroups. With the NETI2 method, the identified networks have 730 and 540 edges for the two groups. For the first group, the networks of the subgroups have 426, 318, and 40 edges. In addition, for the second group, the networks of the subgroups have 228 and 22 edges. More detailed results on grouping and network structures are omitted and available from the authors. It is noted that, as in other unsupervised heterogeneity analyses, it is challenging to conclude which set of analysis results is more sensible.

7. Discussion

The significance of heterogeneity analysis for cancer omics and many other data has been fully established in the literature. In this article, building on and also significantly advancing from the existing literature, we have developed and implemented a new unsupervised heterogeneity analysis approach. The building blocks of the proposed approach, in particular the decomposition of observed data, network-based analysis, and heterogeneity analysis with a nested structure, have been more or less considered in the literature. However, the proposed “combination” is novel, and the methodological, theoretical, and computational developments are highly nontrivial. The extensive simulation and data analysis results have suggested that the proposed approach can provide a practically useful tool for cancer omics (and many other) data.

This study can be potentially extended in multiple ways. The proposed approach has been presented at two levels. Conceptually, it is almost straightforward to extend to multiple levels. The GGM makes the joint normality assumption, which can be relaxed following the existing literature. With the proposed approach, it is assumed that there are two or more levels. When it is observed that the number of subgroups is similar to that of groups or that the subgroups have similar network structures, a warning sign should be raised. It can be of interest to develop testing of multi-level grouping. Last but not least, more data analyses and comparisons of single versus multi-level heterogeneity analyses are desired.

Supplementary Material

The supporting information file contains additional theoretical developments (referenced in Section 3), computational details (referenced in Section 4), additional simulation and real data results (referenced in Sections 5 and 6). The R programs are available in the “MHHA” repository at https://github.com/shuanggema/MHHA. The Pan Cancer Analysis of Whole Genomes (PCAWG) data are openly available from the ICGC Data Portal https://dcc.icgc.org/pcawg. The pathway information is openly available from the KEGG database https://www.kegg.jp/entry/hsa04115. The supporting information is available online at math.scichina.com and link.springer.com. The supporting materials are published as submitted, without typesetting or editing. The responsibility for scientific accuracy and content remains entirely with the authors.