Subgroup Analysis of Differential Networks with Latent Variables

Abstract

Differential network analysis serves as a crucial tool in capturing variations in network rewiring patterns across different biological conditions. Real-world observational data often contain subgroup structures with differing statistical properties, potentially leading to heterogeneity among differential networks. However, existing graphical model-based heterogeneity analysis methods are designed for subgroup networks rather than differential networks. Moreover, these methods require sparsity within each network, which is ineffective for dense networks, especially when observed variables are confounded by latent (unobserved) variables. In this article, we focus on estimating differential networks between an unlabeled heterogeneous group and a baseline group, and develop subgroup analysis from the perspective of differential networks. By imposing a sparse plus low-rank structure on the baseline network and sparsity on differential networks, which characterize and balance the influence of latent variables, the proposed method allows for effective estimation of sparse differential networks and non-sparse subgroup networks. We develop an efficient computational algorithm for this purpose. Simulation studies demonstrate the competitive performance of the proposed approach over closely related alternatives. An application to a set of data on Non-Small Cell Lung Cancer (NSCLC) further confirms the utility of the methodology.

1. Introduction

Network analysis provides key insights into the interactions among components of complex systems, which has become particularly popular in biological studies. Accumulated evidence suggests that interactions among components of biological systems can vastly change over (evolutionary) time, when the system responds to external stimuli, or in disease conditions (Shojaie 2021). For example, variations in gene regulatory network have been observed in cancer research. As a disease primarily driven by genetic alterations, cancer exhibits distinct gene regulatory patterns in different state or microenvironment of cancerous tissues (Tian et al. 2016; Bhuva et al. 2019; Galindez et al. 2024). This rewiring provides crucial insights into the underlying biological mechanisms. While traditional differential expression analysis remains valuable for identifying individual molecular components, differential network analysis provides complementary advantages in characterizing system-level rewiring across biological conditions(Ideker and Krogan 2012; Yuan et al. 2017; Zhang et al. 2024). There is increasing interest in inferring true differential networks to elucidate the complex mechanisms within specific biological contexts.

A fundamental idea to infer the differential network is to separately estimate individual network followed by computation of their topological differences. In the biological context, numerous well-established methodologies exist for constructing gene networks, including matrix decomposition-based models (Kim and Tidor 2003), information theory-based models (Margolin et al. 2006), Gaussian graphical models (GGMs) (Shutta et al. 2022), and more. We focus on the Gaussian graphical models, which are popular due to their ability to capture conditional dependencies among observed variables. The GGM is encoded by the sparsity pattern of the precision matrix, where the non-zero elements of the precision matrix correspond to the edges of the graph. Naturally, the differential network is defined as the differences between the corresponding precision matrices. Many successful regularization techniques are developed for separately estimating single GGMs (Meinshausen and Bühlmann 2006; Yuan and Lin 2007; Friedman et al. 2008; Mohan et al. 2012; Colombi et al. 2024). Compared to separate estimation, joint estimation takes advantage of certain similarities inherent in multiple GGMs to improve the network estimation in each GGM and then better delineate their differences (Guo et al. 2011; Danaher et al. 2014). However, such methods usually shrink both the precision matrices and the differences, their performance is therefore limited if the precision matrices are not sparse. As an alternative, direct estimation of the difference between two GGMs is proposed (Zhao et al. 2014; Yuan et al. 2017). It only assumes that the difference of the precision matrices is sparse, rather than each of the precision matrices. As a natural extension, several works further consider scenarios with joint estimation of multiple differential networks and encouraged common structure in differential networks (Zhang et al. 2017; Ou-Yang et al. 2019). We refer the readers to Shojaie (2021) and Tsai et al. (2022) for nice overviews on differential network estimation. Although the above approaches are useful in particular contexts, they are limited to estimating a single differential network of two homogeneous groups or multiple differential networks with pre-specified subgroups. There is a lack of methods to estimate differential networks from unlabeled heterogeneous population. One of the important application scenarios addressed in this paper is the analysis of cancer populations, which can be further subdivided into different molecular subgroup (referred to as heterogeneity). Our goal is to identify subgroup structures from the perspective of differential networks and simultaneously obtain specific differential networks for each subgroup. In contrast to focusing on the specific networks of each subgroup(referred to as subgroup networks), we aim to reveal the differential networks between subgroups and a specified baseline group(referred to as differential networks), characterizing the potential heterogeneity within the target population from the perspective of differential networks.

Our motivating example is the joint analysis of RNA-sequencing (RNA-seq) data from The Cancer Genome Atlas (TCGA) cancer patients and Genotype Tissue Expression project (GTEx) healthy samples. Such resources provide a great opportunity to gain better insight into complex human diseases (Wang et al. 2018), especially in exploring differences among genetics of cancer patients and healthy samples. The gene interactions in a biological network are overall stable in a particular type of normal human tissue but widely variant in diseased tissues (Chen et al. 2021). This is related to the strong heterogeneity of cancer (Pang et al. 2022), which prompts us to explore multiple differential networks corresponding to different subgroups and the baseline group. It is worth emphasizing that prior knowledge of the subgroup membership may not always be available, the patterns of differential network connections can be predictive of subgroup identification. This provides another reason for simultaneously revealing the topological representation of potential biological mechanisms and capturing the molecular heterogeneity of human cancers without prior knowledge.

Some literature conducts subgroup analysis based on finite Gaussian mixture graphical models (Gao et al. 2016; Hao et al. 2018; Ren et al. 2022). However, these methods are designed for subgroup networks themselves rather than differential networks, which are different from our purpose but provide valuable insights. Moreover, these methods require sparsity within each subgroup network, which is ineffective for dense networks. In many applications, it is clear that relationships between the observed variables are confounded by unobserved, latent factors. For example, for gene network estimation, besides observed genes, there also exist some unobserved molecules or factors (e.g., MicroRNA) in the cell that participate in the process of gene regulation (Beal et al. 2005). Another source of confounding that motivates us is the pre-selection of genes in data analysis, which can lead to the loss of information, thereby limiting a comprehensive understanding of the overall data structure. Due to confounding, the marginal precision matrix is not sparse, whereas the traditional approaches of estimating sparse Gaussian graphical models are of limited effect. To address this problem, latent variable Gaussian graphical models (LVGGMs) are proposed by Chandrasekaran et al. (2012), which exploits the observation that the marginal graph of the observed variables can be decomposed into a superposition of a sparse matrix and a low-rank matrix. This decomposition can be particularly useful when the data is corrupted by latent confounders, and some achievements have been made in the past decade (Xu et al. 2017; Na et al. 2021; Li et al. 2023; Wang et al. 2023).

Based on the above discussions, we propose a regularized graph learning approach to reconstruct differential networks between an unlabeled heterogeneous group and a baseline group, which simultaneously incorporates the non-sparsity confounded by latent variables. A diagram illustration of the proposed model is shown in Figure 1. The observations obtained consist of data from the baseline group and the unlabeled heterogeneous group. The differential network is constructed by the edge-wise subtraction of the strengths between the subgroup networks and baseline network. Due to the confounding effect of latent variables, both differential networks and subgroup networks may be non-sparse, while their differences exhibit sparsity. Borrowing from recent developments in finite mixture of Gaussian graphical models, we develop a subgroup analysis method of differential networks with latent variables under the framework of penalized likelihood. For feasibility reasons, we take a novel strategy that imposes a sparse plus low-rank structure on the baseline network, and sparsity on differential networks, which characterize and balance the influence of latent variables. This study may advance existing research in effective estimation of differential networks with unsupervised (unlabeled) heterogeneity and non-sparse subgroup networks. Although initially inspired by gene expression data analysis, the methodology demonstrates broad applicability across diverse omics modalities, including transcriptomics (RNA-seq, microarray), proteomics, and metabolomics. The framework is well-suited for comparative heterogeneous network analysis among diseases and normal states, or between control groups and target heterogeneous cohorts. Hence, it can also be employed to other system-level investigations. For example, in neuroscience, it can identify connectivity differences between task-state and resting-state functional brain networks.

Fig. 1.

Illustration of the proposed method. Due to the influence of unobservable latent variables(dashed edges), the marginal graph (solid red and blue edges) may be dense, while their differences exhibit sparsity

This article extends the differential network analysis to unlabeled heterogeneous situation and allows for effective estimation of sparse differential networks and non-sparse subgroup networks. The rest of this article is organized as follows. In Section 2, the proposed subgroup analysis method is described. Section 3 presents an elaborate simulation study, where the performance of the proposed method compared to the state-of-the-art models is evaluated. An application of the new method to joint analysis of TCGA and GTEx data is given in Section 4. Finally, we conclude in Section 5 with a discussion. Additional algorithm details and numerical experiments are provided in Appendix A and Appendix B.

2. Methodology

2.1. Notation and Model Framework

Denote $n$ as the number of independent observations $\left\{{\mathit{x}}_i,y_i\right\},i=1,\dots ,n$, where $y_i\in \{0,1\}$ indicates whether samples $i$ belongs to heterogeneous group, i.e., $y_i=1$ if the $i$-th sample is from heterogeneous group, otherwise $y_i=0$. For all observations, $p$-dimensional measurement ${\mathit{x}}_i$ is available, such as the expression levels of $p$ genes. Let $n_0$ and $n_1=n-n_0$ denote the number of samples in the baseline and heterogeneous groups, respectively. We assume that the baseline group follows a single Gaussian distribution, while the heterogeneous group follows a mixture of $K^{*}$ Gaussian distributions, where $K^{*}$ may be unknown. Specifically, $n_0$ samples of baseline group follow a Gaussian distribution with marginal covariance matrix ${𝚺}_0^{*}$:

$$\begin{gathered} f_0\left(\mathit{x};{\mathit{\mu }}_0^{*},{𝚺}_0^{*}\right)=(2\pi {)}^{-p/2}{\left|{𝚺}_0^{*}\right|}^{-1/2} \\ \text{exp}\left(-\frac{1}{2}{\left(\mathit{x}-{\mathit{\mu }}_0^{*}\right)}^{\top }{\left({𝚺}_0^{*}\right)}^{-1}\left(\mathit{x}-{\mathit{\mu }}_0^{*}\right)\right). \end{gathered}$$

The corresponding precision matrix is denoted as ${𝚯}_0^{*}={\left({𝚺}_0^{*}\right)}^{-1}$. To characterize the effect of latent variables, we impose LVGGM to model the network of baseline group, which leads to much milder assumption because ${𝚯}_0^{*}$ may be non-sparse. Under the framework of LVGGM, the precision matrix ${𝚯}_0^{*}$ can be represented as the sum of a sparse matrix ${\mathit{S}}_0^{*}$ and a low-rank matrix ${\mathit{L}}_0^{*}$, i.e., ${𝚯}_0^{*}={\mathit{L}}_0^{*}+{\mathit{S}}_0^{*}$, and $\text{rank}\left({\mathit{L}}_0^{*}\right)=r$.

Considering the unknown subgroup structures of unlabeled heterogeneous group, we further assume that the $n_1$ samples can be divided into $K^{*}$ subgroups. Samples in the $k$-th subgroup come from the following Gaussian distribution:

$$\begin{gathered} f_k\left(\mathit{x};{\mathit{\mu }}_k^{*},{𝚺}_k^{*}\right)=(2\pi {)}^{-p/2}{\left|{𝚺}_k^{*}\right|}^{-1/2} \\ \text{exp}\left(-\frac{1}{2}{\left(\mathit{x}-{\mathit{\mu }}_k^{*}\right)}^{\top }{\left({𝚺}_k^{*}\right)}^{-1}\left(\mathit{x}-{\mathit{\mu }}_k^{*}\right)\right). \end{gathered}$$

Let the mixture proportions of the $K^{*}$ subgroups be ${\mathit{\pi }}^{*}=\left({\pi }_1^{*},\dots ,{\pi }_{K^{*}}^{*}\right)$, satisfying $\sum _{k=1}^{K^{*}}{\pi }_k^{*}=1$. The structural information of the heterogeneous group can be characterized by the following mixture distribution:

$$f\left(\mathit{x}\right)=\sum _{k=1}^{K^{*}}{\pi }_k^{*}{f}_k\left(\mathit{x};{\mathit{\mu }}_k^{*},{𝚺}_k^{*}\right).$$

The precision matrix is denoted as ${𝚯}_k^{*}={\left({\Sigma }_k^{*}\right)}^{-1},k=1,\dots ,K^{*}$. The differential networks are constructed by the edge-wise subtraction of the strengths between the subgroup networks and baseline network, defined as ${𝚫}_k^{*}={𝚯}_k^{*}-{𝚯}_0^{*},k=1,\dots ,K^{*}$. Inspired by the research of Yuan et al. (2017), we assume that the differential networks ${𝚫}_k^{*}$ are sparse, which allows for non-sparsity of subgroup networks. In our context, it helps to balance the influence of latent variables and yield interpretable differential networks.

2.2. Proposed Method

Denote $\mathit{\mu }=\left({\mathit{\mu }}_1,\dots ,{\mathit{\mu }}_K\right),𝚯=\left({𝚯}_1,\dots ,{𝚯}_K\right),\mathit{\pi }=\left({\pi }_1,\dots ,{\pi }_K\right)$, and the set of differential networks $𝚫=\left({𝚫}_1,\dots ,{𝚫}_K\right)$. Based on the previous analysis, we have ${𝚯}_k={\mathit{S}}_0+{\mathit{L}}_0+{𝚫}_k$. To reconstruct differential networks and incorporates the non-sparsity confounded by latent variables, we propose to minimize the following penalized negative log-likelihood function for the observed data from two groups:

$$\begin{gathered} ℒ\left({\mathit{S}}_0,{\mathit{L}}_0,\mathit{\mu },𝚫,\mathit{\pi }\mid \mathit{X}\right)≔-\frac{1}{2}\text{log}\text{det}\left({\mathit{L}}_0+{\mathit{S}}_0\right) \\ +\frac{1}{2}\text{tr}\left({\widehat{𝚺}}_0\left({\mathit{L}}_0+{\mathit{S}}_0\right)\right)-\frac{1}{n_1}\sum _{i:y_i=1}\text{log}\left(\sum _{k=1}^K{\pi }_k\times f_k\left({\mathit{x}}_i;{\mathit{\mu }}_k,{\left({\mathit{L}}_0+{\mathit{S}}_0+{𝚫}_k\right)}^{-1}\right)\right)+𝒫\left({\mathit{S}}_0,\mathit{\mu },𝚫\right), \end{gathered}$$ (1)

where $\mathit{X}$ denotes the collection of all observed data and ${\widehat{𝚺}}_0=\frac{1}{n_0}\sum _{i:y_i=0}{\mathit{x}}_i{\mathit{x}}_i^{\top }$ is sample covariance matrix for baseline group. The non-penalized part of the objective function comprises the negative log-likelihood (up to a constant) of samples from the baseline group and heterogeneous group. Regarding the form of penalty function, to reduce computational complexity and induce low-rank structure of ${\mathit{L}}_0$, we adopt the reparameterization strategy ${\mathit{L}}_0=-{\mathit{U}}_0{\mathit{U}}_0^{\top },{\mathit{U}}_0\in {\mathbf{R}}^{p\times r}$ via matrix factorization techniques (Xu et al. 2017). Then $-{\mathit{L}}_0$ is guaranteed to be a positive semidefinite matrix with $\text{rank}\left({\mathit{L}}_0\right)\le r$. To enforce other constraints on networks, we take the penalty function has the following form:

$$\begin{gathered} 𝒫\left({\mathit{S}}_0,\mathit{\mu },𝚫\right)=\sum _{i\ne j}p\left(\left|{\mathit{S}}_{0ij}\right|;{\lambda }_1\right)+\sum _{k=1}^K\sum _{i,j}p\left(\left|{𝚫}_{kij}\right|;{\lambda }_2\right) \\ +\sum _{k<k^{\prime }}p\left({\left({‖{\mathit{\mu }}_k-{\mathit{\mu }}_{k^{\prime }}‖}_2^2+{‖{𝚫}_k-{𝚫}_{k^{\prime }}‖}_F^2\right)}^{1/2};{\lambda }_3\right), \end{gathered}$$ (2)

where $\Vert \cdot {\Vert }_F$ represents the Frobenius norm, $p(\cdot ;\lambda )$ is a non-convex penalty function with tuning parameter $\lambda >0$, and $K$ is a given constant satisfying $K>K^{*}$. In numerical studies, we employ the minimax concave penalty (MCP) proposed by Zhang (2010), and other penalties such as smoothly clipped absolute deviation (SCAD) (Fan and Li 2001) may also be applicable. Denote the minimizer of (1) as $({\widehat{\mathit{S}}}_0,{\widehat{\mathit{L}}}_0,\widehat{\mathit{\mu }},\widehat{𝚫},\widehat{\mathit{\pi }})$. Let $({\widehat{{\rm Y}}}_1,\dots ,{\widehat{{\rm Y}}}_{\widehat{K}})$ represent the unique values of $({\widehat{𝛀}}_1,\dots ,{\widehat{𝛀}}_K)$ with ${\widehat{𝛀}}_k=\left({\widehat{\mathit{\mu }}}_k,{\widehat{𝚫}}_k\right)$. These unique values induce a partition of $\{1,\dots ,K\}$ with ${\left\{k:{\widehat{𝛀}}_k\equiv {\widehat{{\rm Y}}}_l,k=1,\dots ,K\right\}}_{l=1,\dots ,\widehat{K}}$. Consequently, this partition results in $\widehat{K}$ distinct subgroups. The mixture probabilities associated with these subgroups can be derived by aggregating the corresponding entries of the estimated probability vector $\widehat{\mathit{\pi }}$.

The proposed method integrates three strategically designed penalties to achieve comprehensive regularization. The first two penalty terms in (2) correspond to the conditional sparsity of LVGGM (Chandrasekaran et al. 2012) and differential sparsity of differential networks, respectively. The third term introduces a fusion penalty strategy effectively compressing the initial value of $K$ through data-driven identification of subgroup numbers. Different from the fusion strategy of Ren et al. (2022), we conduct subgroup analysis from the perspective of differential networks, rather than the network itself. Compared with the work of Na et al. (2021), we impose low-rank plus sparse decomposition for the baseline network and sparsity for each differential network, rather than imposing low-rank plus sparse decomposition for each differential network. In addition, our differential network analysis is designed for unsupervised heterogeneity. The proposed method can be interpreted as a joint estimation framework that combines matrix factorization to jointly estimate baseline group networks and multi-group differential networks. By leveraging cross-group information sharing, the joint estimation strategy improves parameter estimation.

2.3. Model Estimation

To solve the optimization problem (1), we develop a block coordinate descent algorithm for optimizing the reparameterized objective function as Algorithm 1 described, which involves updating the estimation of $({\mathit{U}}_0,{\mathit{S}}_0)$ and $(\mathit{\mu },𝚫,\mathit{\pi })$ iteratively while holding the others fixed until convergence. Convergence is reached when the difference of loss function between two consecutive estimates falls below a pre-defined threshold (taken as 0.01 in our numerical study). We adopt the same initialization strategy of Xu et al. (2017). Given ${\mathit{U}}_0^{(t-1)},{\mathit{S}}_0^{(t-1)}$ and ${\mathit{L}}_0^{(t-1)}=-{\mathit{U}}_0^{(t-1)}{\mathit{U}}_0^{(t-1{)}^{\top }}$, the update for ${\mathit{\mu }}^{\left(t\right)},{𝚫}^{\left(t\right)},{\mathit{\pi }}^{\left(t\right)}$ can be written as minimizing

$$\begin{gathered} {\ell }_1(\mathit{\mu },𝚫,\mathit{\pi }\mid \mathit{X})≔-\frac{1}{n_1}\sum _{i:y_i=1}\text{log}\left(\sum _{k=1}^K{\pi }_k\right.\left.\times f_k\left({\mathit{x}}_i;{\mathit{\mu }}_k,{\left({\mathit{L}}_0^{(t-1)}+{\mathit{S}}_0^{(t-1)}+{𝚫}_k\right)}^{-1}\right)\right) \\ +\sum _{k=1}^K\sum _{i,j}p\left(\left|{𝚫}_{kij}\right|;{\lambda }_2\right)+\sum _{k<k^{\prime }}p\left(\left({‖{\mathit{\mu }}_k-{\mathit{\mu }}_{k^{\prime }}‖}_2^2\right.\right.\left.{\left.+{‖{𝚫}_k-{𝚫}_{k^{\prime }}‖}_F^2\right)}^{1/2};{\lambda }_3\right). \end{gathered}$$ (3)

We apply the Expectation-Maximum(EM) and Alternating Direction Method of Multipliers (ADMM) algorithm for optimizing (3). Given fixed ${\mathit{U}}_0^{(t-1)},{\mathit{S}}_0^{(t-1)}$, the process of solving ${𝚯}_k$ or ${𝚫}_k$ is equivalent, because the difference between them is fixed as ${𝚯}_0^{(t-1)}$. For simplicity of notation, we rewrite the sub-objective function to solve ${𝚯}_k$ instead of ${𝚫}_k$, and compute ${𝚫}_k$ by ${𝚫}_k={𝚯}_k-{𝚯}_0^{(t-1)}$ as the final result. Then (3) is equivalent to

$${\tilde{\ell }}_1(\mathit{\mu },𝚯,\mathit{\pi }\mid \mathit{X})≔-\frac{1}{n_1}\sum _{i:y_i=1}\text{log}\left(\sum _{k=1}^K{\pi }_k\times f_k\left({\mathit{x}}_i;{\mathit{\mu }}_k,{𝚯}_k^{-1}\right)\right)+𝒫(\mathit{\mu },𝚯),$$

where

$$𝒫(\mathit{\mu },𝚯)=\sum _{k=1}^K\sum _{i,j}p\left(\left|{𝚯}_{kij}-{𝚯}_{0ij}^{(t-1)}\right|;{\lambda }_2\right)+\sum _{k<k^{\prime }}p\left({\left({‖{\mathit{\mu }}_k-{\mathit{\mu }}_{k^{\prime }}‖}_2^2+{‖{𝚯}_k-{𝚯}_{k^{\prime }}‖}_F^2\right)}^{1/2};{\lambda }_3\right).$$

In the $m$–step of the EM algorithm, the following penalized log-likelihood function for the complete data needs to be minimized:

$$\begin{gathered} E_{\gamma \mid \mathit{X},\mathit{\mu },𝚯}\left[{\tilde{\ell }}_1(\mathit{\mu },𝚯,\mathit{\pi }\mid \mathit{X},\mathit{\gamma })\right]=-\frac{1}{n_1}\sum _{i:y_i=1}\sum _{k=1}^K \\ {\gamma }_{ik}^{\left(m\right)}\left(\text{log}{\pi }_k+\text{log}{f}_k\left({\mathit{x}}_i;{\mathit{\mu }}_k,{𝚯}_k^{-1}\right)\right)+𝒫\left(\mathit{\mu },𝚯\right), \end{gathered}$$ (4)

where ${\gamma }_{ik}^{\left(m\right)}$ can be computed based on ${\pi }_k^{(m-1)},{\mathit{\mu }}_k^{(m-1)}$, and ${𝚯}_k^{(m-1)}$ obtained in the previous iteration, that is,

$${\gamma }_{ik}^{\left(m\right)}=\frac{{\pi }_k^{(m-1)}{f}_k\left({\mathit{x}}_i;{\mathit{\mu }}_k^{(m-1)},{\left({𝚯}_k^{(m-1)}\right)}^{-1}\right)}{\sum _{k=1}^K{\pi }_k^{(m-1)}{f}_k\left({\mathit{x}}_i;{\mathit{\mu }}_k^{(m-1)},{\left({𝚯}_k^{(m-1)}\right)}^{-1}\right)}.$$ (5)

Given ${\gamma }_{ik}^{\left(m\right)}$, the maximization step minimizing (4) with respect to $(\mathit{\mu },𝚯,\mathit{\pi })$, which yields the update of parameters. More calculation details can be found in the Appendix B.

Given ${\mathit{\mu }}^{\left(t\right)},{𝚫}^{\left(t\right)},{\mathit{\pi }}^{\left(t\right)}$, the update for ${\mathit{U}}_0^{\left(t\right)},{\mathit{S}}_0^{\left(t\right)}$ is the minimizer of

$$\begin{gathered} {\ell }_2\left({\mathit{U}}_0,{\mathit{S}}_0\mid \mathit{X}\right)≔-\frac{1}{2}\text{log}\text{det}\left(-{\mathit{U}}_0{\mathit{U}}_0^{\top }+{\mathit{S}}_0\right) \\ +\frac{1}{2}\text{tr}\left({\widehat{𝚺}}_0\left(-{\mathit{U}}_0{\mathit{U}}_0^{\top }+{\mathit{S}}_0\right)\right)-\frac{1}{n_1}\sum _{i:y_i=1}\text{log} \\ \left(\sum _{k=1}^K{\pi }_k^{\left(t\right)}{f}_k\left({\mathit{x}}_i;{\mathit{\mu }}_k^{\left(t\right)},{\left(-{\mathit{U}}_0{\mathit{U}}_0^{\top }+{\mathit{S}}_0+{𝚫}_k^{\left(t\right)}\right)}^{-1}\right)\right) \\ +\sum _{i\ne j}p\left(\left|{\mathit{S}}_{0ij}\right|;{\lambda }_1\right). \end{gathered}$$ (6)

The sub-objective function (6) can be solved by alternating proximal gradient descent algorithm, which performs one-step gradient descent for ${\mathit{U}}_0$ and ${\mathit{S}}_0$ alternately as described in Algorithm 1. Denote ${𝚯}_0^{(s-1)}=-{\mathit{U}}_0^{(s-1)}{\mathit{U}}_0^{\top (s-1)}+{\mathit{S}}_0^{(s-1)}$, the gradient for ${\mathit{S}}_0,{\mathit{U}}_0$ is

$$\begin{gathered} {\nabla }_{{\mathit{S}}_0}=-\frac{1}{2}{\left({𝚯}_0^{(s-1)}\right)}^{-1}+\frac{1}{2}{\widehat{𝚺}}_0 \\ -\frac{1}{n_1}\sum _{i:y_i=1}\sum _{k=1}^K{\gamma }_{ik}\left({𝚯}_0^{(s-1)}+{\mathit{C}}_{ik}^{\left(t\right)}\right), \\ {\nabla }_{{\mathit{U}}_0}={\nabla }_{{\mathit{S}}_0}\times \left(-2{\mathit{U}}_0^{(s-1)}\right), \\ {\mathit{C}}_{ik}^{\left(t\right)}={𝚫}_k^{\left(t\right)}-\left({\mathit{x}}_i-{\mathit{\mu }}_k^{\left(t\right)}\right){\left({\mathit{x}}_i-{\mathit{\mu }}_k^{\left(t\right)}\right)}^{\top }. \end{gathered}$$

For convenience, the step sizes $\alpha$ and ${\alpha }^{\prime }$ are taken as 0.01 in our numerical study. ${\mathscr{H}}_{{\lambda }_1}(\cdot )$ is the proximal operator for MCP penalty with tuning parameter ${\lambda }_1$, defined as

$${\mathscr{H}}_{{\lambda }_1}(c)=\left\{\begin{array}{ll}\frac{S{T}_{{\lambda }_1}(c)}{(1-1/a)}& \text{if}\left|c\right|⩽a{\lambda }_1,\\ c& \text{if}\left|c\right|>a{\lambda }_1.\end{array}\right.$$

where $S{T}_{{\lambda }_1}(\cdot )$ is the soft thresholding operator, $a$ is taken as 3 which controls the concavity of the MCP penalty function.

Algorithm 1.

Initialize $\left({\mathit{U}}_0^{\left(0\right)},{\mathit{S}}_0^{\left(0\right)}\right)$

Repeat for $t=1,2,3,\dots$

1. Given ${\mathit{U}}_0^{(t-1)},{\mathit{S}}_0^{(t-1)}$, update ${\mathit{\mu }}^{\left(t\right)},{𝚫}^{\left(t\right)},{\mathit{\pi }}^{\left(t\right)}$ as the minimizer of (3).

Repeat for $m=1,2,3,\dots$ as follows:

(1) E-step: Update the subpopulation assignment ${\gamma }_{ik}^{\left(m\right)}$ using (5)

(2) M-step: Update ${\mathit{\mu }}^{\left(m\right)},{𝚫}^{\left(m\right)},{\mathit{\pi }}^{\left(m\right)}$ (details are provided in Appendix B).

2. Given ${\mathit{\mu }}^{\left(t\right)},{𝚫}^{\left(t\right)},{\mathit{\pi }}^{\left(t\right)}$, update ${\mathit{U}}_0^{\left(t\right)},{\mathit{S}}_0^{\left(t\right)}$ as the minimizer of (6).

Repeat for $s=1,2,3,\dots$ as follows:

(1) Update ${\mathit{S}}_0^{\left(s\right)}$:

${\mathit{S}}_0^{\left(s\right)}={\mathscr{H}}_{{\lambda }_1}\left({\mathit{S}}_0^{(s-1)}-\alpha {\nabla }_{{\mathit{S}}_0}{\ell }_2\left({\mathit{U}}_0^{(s-1)},{\mathit{S}}_0^{(s-1)}\right)\right)$.

(2) Update ${\mathit{U}}_0^{\left(s\right)}$:

${\mathit{U}}_0^{\left(s\right)}={\mathit{U}}_0^{(s-1)}-{\alpha }^{\prime }{\nabla }_{{\mathit{U}}_0}{\ell }_2\left({\mathit{U}}_0^{(s-1)},{\mathit{S}}_0^{(s-1)}\right)$.

3. Repeat Step 1 and 2 until convergence.

2.4. Selection of Tuning Parameters

The proposed approach involves three tuning parameters $\Lambda ≔\left({\lambda }_1,{\lambda }_2,{\lambda }_3\right)$. In the high-dimensional scenario, it becomes imperative to consider computational efficiency when exploring a grid of ${\lambda }_1\times {\lambda }_2\times {\lambda }_3$ penalty parameter combinations. Unless employing a coarse grid structure for these parameters or dedicating considerable time to pinpointing the optimal set, it is advisable to opt for information criteria over computationally intensive techniques like cross-validation. To mitigate computational burdens, we follow a line search approach for tuning parameter selection, as suggested in Hao et al. (2018). In detail, we fix ${\lambda }_2$ and ${\lambda }_3$ at their median value of the given range and conduct a grid search over ${\lambda }_1$. Then with tuned ${\lambda }_1$ and median value of ${\lambda }_3$, we conduct a grid search over ${\lambda }_2$. The line search for ${\lambda }_3$ is the same. In our framework, the tuning parameters are selected through the following AIC-type selection criterion:

$$\begin{gathered} \text{AIC}\left(\Lambda \right)=-\text{log}\text{det}\left(-{\widehat{\mathit{U}}}_0{\widehat{\mathit{U}}}_0^{\top }+{\widehat{\mathit{S}}}_0\right) \\ +\text{tr}\left({\widehat{𝚺}}_0\left(-{\widehat{\mathit{U}}}_0{\widehat{\mathit{U}}}_0^{\top }+{\widehat{\mathit{S}}}_0\right)\right) \\ -\frac{2}{n_1}\sum _{i:y_i=1}\text{log}\left(\sum _{k=1}^{\widehat{K}}{\pi }_k\right.\left.\times f\left({\mathit{x}}_i;{\widehat{\mathit{\mu }}}_k,{\left(-{\widehat{\mathit{U}}}_0{\widehat{\mathit{U}}}_0^{\top }+{\widehat{\mathit{S}}}_0+{\widehat{𝚫}}_k\right)}^{-1}\right)\right) \\ +\frac{2}{n_0}\left(pr+s_0\right)+\sum _{k=1}^{\widehat{K}}\frac{2}{n_1}\left(s_{1k}+s_{2k}\right), \end{gathered}$$ (7)

where $s_0$ is the number of non-zero elements in ${\widehat{\mathit{S}}}_0,s_{1k},s_{2k}$ is the number of non-zero elements in ${\widehat{\mathit{\mu }}}_k$ and ${\widehat{𝚫}}_k$ respectively.

3. Simulation

The proposed methods are tested on several synthetic data in various situations. In our simulation settings, the true number of subgroups is set to $K^{*}=3$, the number of variables is set as $p=100$. We fix the sample size for the heterogeneous group at $n_1=1500$. The observations are generated as follows:

• For samples of baseline group: Firstly, generate a randomly sparse symmetric positive definite matrix $\widetilde{𝚯}\in {\mathbf{R}}^{(p+r)\times (p+r)}$ with sparsity $s$, which means that the number of connections in the network is approximately $s\times (p+r{)}^2$. Specifically, generate an upper triangular matrix ${\mathit{A}}_0$, where the upper triangular elements are non-zero with a probability of $s/2$. The non-zero values are generated from a uniform distribution in the range [−1, 1]. Let the lower triangular elements of this matrix be equal to the upper triangular elements, resulting in the symmetric matrix ${\tilde{\mathit{A}}}_0$. To further obtain a positive definite matrix, set $\widetilde{𝚯}={\tilde{\mathit{A}}}_0+(0.1+\left.\left|{\lambda }_{\mathit{\text{min}}}\left({\tilde{\mathit{A}}}_0\right)\right|\right){\mathit{I}}_{p+r}$, where ${\lambda }_{\mathit{\text{min}}}(\cdot )$ denotes the minimum eigenvalue of the matrix. ${\mathit{I}}_{p+r}$ is the $(p+r)\times (p+r)$ identity matrix. Based on the conclusions from the LVGGM, we calculate its sparse component ${\mathit{S}}_0^{*}≔{\widetilde{𝚯}}_{1:p;1:p}$. The low-rank component ${\mathit{L}}_0^{*}≔-{\widetilde{𝚯}}_{1:p;(p+1):(p+r)}\times {\left({\widetilde{𝚯}}_{(p+1):(p+r);(p+1):(p+r)}\right)}^{-1}\times {\widetilde{𝚯}}_{(p+1):(p+r);1:p}$. Consequently, we generate $n_0$ samples from $𝒩\left(0,{\left({𝚯}_0^{*}\right)}^{-1}\right)$, where ${𝚯}_0^{*}={\mathit{S}}_0^{*}+{\mathit{L}}_0^{*}$ is the true precision matrix corresponding to the observed samples of baseline group.

• For samples of heterogeneous group: Initially generate $K^{*}$ random sparse symmetric matrices ${\mathit{D}}_k\in {\mathbf{R}}^{p\times p}$ (consistent with the generation method for the basaline network), with sparsity level $s$. We obtain ${𝚯}_k={\mathit{D}}_k+{𝚯}_0^{*}$. To ensure the positive definiteness of the true precision matrix ${𝚯}_k^{*}$, we set ${𝚯}_k^{*}={𝚯}_k+(0.1+\left.\left|{\lambda }_{\mathit{\text{min}}}\left({𝚯}_k\right)\right|\right){\mathit{I}}_p$. Consequently, we have ${𝚫}_k^{*}={𝚯}_k^{*}-{𝚯}_0^{*}$. The first 8 components of the mean vector ${\mathit{\mu }}_k^{*}$ are set to ${\left(\mu 1_4^{\top },-\mu 1_4^{\top }\right)}^{\top }I\left({\eta }_i=1\right)+\mu 1_{8}I\left({\eta }_i=2\right)+(-\mu )1_{8}I\left({\eta }_i=3\right)$, where $1_4$ is a vector of length 4 with all components equal to 1. The remaining $p-8$ components are set to zero. Set the mixture probabilities ${\mathit{\pi }}^{*}=\left({\pi }_1^{*},{\pi }_2^{*},{\pi }_3^{*}\right)$, the subgroup labels ${\eta }_i$ are randomly sampled from {1, 2, 3} according to mixture probabilities. Subsequently, based on the value of ${\eta }_i$, generate samples from the corresponding subgroup distribution: ${\mathit{x}}_i~𝒩\left({\mathit{\mu }}_{{\eta }_i}^{*},{\left({\Theta }_{{\eta }_i}^{*}\right)}^{-1}\right)$.

We comprehensively observe and evaluate the performance of the proposed method under different scenarios, including different values for $r,s,n_0,{\mathit{\mu }}^{*}$ and ${\mathit{\pi }}^{*}$. With the proposed approach, we set the rank of the low-rank matrix to the true value $r$. Informal exploration of various values of $K>3$ yield consistent results, prompting the decision in this study to set a fixed $K=6$. To comprehensively evaluate our method, we consider the following competing methods:

• Separate: When updating the mixture distribution parameters of the heterogeneous part, the precision matrix information of the baseline group is not considered. This is similar to the approach by Ren et al. (2022), which is based on the sparse assumption of precision matrix. To obtain the differential networks, we directly subtract the baseline network estimated by Xu et al. (2017).

• SCAN: We apply the simultaneous clustering and estimation of heterogeneous graphical models (SCAN) (Hao et al. 2018) method to estimate the networks of the heterogeneous group, which requires the subgroup number is equal to the true value. We consider both cases of $K=6$ and $K=3$ in the comparison, representing scenarios of incorrectly and correctly specified subgroup number, respectively. The differential networks are estimated similar to Separate method, which subtracts the baseline network from the heterogeneous networks directly.

• K-means+JGL: The Joint Graphical Lasso (Danaher et al. 2014) method also considers the presence of similar structures among multiple correlated graphs but requires known subgroup labels. We first use the K-means algorithm to obtain sample subgroup labels and then apply the Joint Graphical Lasso method to estimate precision matrices. This methodology is implemented using the R packages NbClust and JGL. Similarly, we consider cases of $K=6$ and $K=3$. We apply K-means+JGL to estimate the network structure of the heterogeneous group. The calculation of the differential networks follows the same procedure as described for the Separate and SCAN method.

• K-means+D-trace: The method of Yuan et al. (2017) is based on the D-trace loss and directly estimates the differential network by applying a sparse penalty to the differential precision matrix without estimating the precision matrix itself. Unlike Separate, SCAN, and K-means+JGL, this method does not require estimating the precision matrices. Given the problem setting in this paper, we first obtain sample subgroup labels through the K-means algorithm and then train differential networks. Similarly, we consider both $K=6$ and $K=3$ scenarios.

Among the competing methods, Separate involves three hyperparameters, SCAN and JGL operates with two regularization parameters (with SCAN’s mean penalty removed for fairness of comparison) and D-trace requires only a single tuning parameter. We systematically employ the AIC criterion for tuning parameters selection across all competing methods, and conduct a line search to determine the optimal parameters configurations (except for D-race, which involves only one tuning parameter). Figure 2 presents a simple example with ten nodes to illustrate the effectiveness of proposed method compared to other alternative methods. In this toy example, both the Proposed method and the Separate method accurately identify the true number of subgroups $(K^{*}=3)$ with an initial value $K=6$. Intuitively, the method pro-posed performs better in achieving effective estimation of both sparse differential networks and non-sparse subgroup networks. To numerically gauge performance of the proposed and alternative approaches, we consider the following measures:

• Estimate errors of low-rank and sparse components and precision matrices of the baseline group in terms of Frobenius norm. Denote as ${‖{\widehat{\mathit{S}}}_0-{\mathit{S}}_0^{*}‖}_F,{‖{\widehat{\mathit{L}}}_0-{\mathit{L}}_0^{*}‖}_F$ and ${‖{\widehat{𝚯}}_0-{𝚯}_0^{*}‖}_F$, where ${\widehat{\mathit{L}}}_0=-{\widehat{\mathit{U}}}_0{\widehat{\mathit{U}}}_0^{\top }$.

• Proportion of $\widehat{K}=K^{*}$ (denoted as Per).

• Mean and standard deviation of $\widehat{K}$.

• Clustering error of subgroup division, defined as

  $$\begin{gathered} \text{CE}(\widehat{\phi },\phi )≔{\left(\genfrac{}{}{0pt}{}{n}{2}\right)}^{-1}\times \mid \left\{(i,j):I\left(\widehat{\phi }\left({\mathit{x}}_i\right)=\widehat{\phi }\left({\mathit{x}}_j\right)\right)\right. \\ \left.\ne I\left(\phi \left({\mathit{x}}_i\right)=\phi \left({\mathit{x}}_j\right)\right);i<j\right\}\mid , \end{gathered}$$

  where $\widehat{\phi }$ and $\phi$ are the estimated and true subgroup divisions.

• Root Mean Squared Error (RMSE) of $\mathit{\mu }$ and $𝚫$. When $\widehat{K}=K^{*}$, RMSE is defined as $\sum _{k=1}^{K^{*}}{‖{\widehat{\mathit{\mu }}}_k-{\mathit{\mu }}_k^{*}‖}_2/K^{*},\sum _{k=1}^{K^{*}}{‖{\widehat{𝚫}}_k-{𝚫}_k^{*}‖}_F/K^{*}$. When $\widehat{K}\ne K^{*}$, RMSE is defined as

  $$\begin{gathered} \text{RMSE}\left(\mathit{\mu }\right)=\frac{1}{\widehat{K}}\sum _{k=1}^{\widehat{K}}\sum _{l^{\prime }=1}^{K^{*}}{‖{\widehat{\mathit{\mu }}}_k-{\mathit{\mu }}_{l^{\prime }}^{*}‖}_2 \\ \times I\left(l^{\prime }=\underset{l}{\text{argmin}}\left\{{‖{\widehat{\mathit{\mu }}}_k-{\mathit{\mu }}_l^{*}‖}_2^2+{‖{\widehat{𝚫}}_k-{𝚫}_l^{*}‖}_F^2\right\}\right), \\ \text{RMSE}\left(𝚫\right)=\frac{1}{\widehat{K}}\sum _{k=1}^{\widehat{K}}\sum _{l^{\prime }=1}^{K^{*}}{‖{\widehat{𝚫}}_k-{𝚫}_{l^{\prime }}^{*}‖}_F \\ \times I\left(l^{\prime }=\underset{l}{\text{argmin}}\left\{{‖{\widehat{\mathit{\mu }}}_k-{\mathit{\mu }}_l^{*}‖}_2^2+{‖{\widehat{𝚫}}_k-{𝚫}_l^{*}‖}_F^2\right\}\right). \end{gathered}$$
• True/False Positive Rates (TPR/FPR) for the identification of non-zero elements in the off-diagonal of the differential precision matrix. When $\widehat{K}=K^{*}$, defined as

  $$\begin{gathered} \text{TPR}=\frac{\sum _{k=1}^{K^{*}}\sum _{i<j}I\left({𝚫}_{kij}\ne 0,{\widehat{𝚫}}_{kij}\ne 0\right)}{\sum _{k=1}^{K^{*}}\sum _{i<j}I\left({𝚫}_{kij}\ne 0\right)}, \\ \text{FPR}=\frac{\sum _{k=1}^{K^{*}}\sum _{i<j}I\left({𝚫}_{kij}=0,{\widehat{𝚫}}_{kij}\ne 0\right)}{\sum _{k=1}^{K^{*}}\sum _{i<j}I\left({𝚫}_{kij}=0\right)}. \end{gathered}$$

  When $\widehat{K}\ne K^{*}$, the definitions are modified accordingly, similar to the RMSE definition.

Fig. 2.

An example illustrating network reconstruction results of the proposed method and other alternative methods. Red and blue edges represent correctly inferred interactions of differential networks and subgroup networks, while gray edges indicate incorrectly inferred interactions

With 100 replicates, summary results of Simulation 1 under the setting of $r=2,s=0.02,\mu =1.5$ with balanced and imbalanced designs are shown in Table 1. Results for other combinations of $r=5,s=0.05,\mu =2$ are shown in tables of the AppendixA. Observations made under different settings are similar. For the result of Simulation 1, the proposed method correctly identifies the number of subgroups at a high rate, outperforming the subgroup identification rate under separate estimation. For example, in a fully balanced setting with $n_0=1500$ and ${\mathit{\pi }}^{*}=(1/3,1/3,1/3)$, the proposed method correctly identifies the number of subgroups with a rate of 93%, while the separate estimation achieves a correct identification rate of 27%. Compared to the separate estimation method, the proposed method improves the ability to estimate the baseline group network. This improvement is reflected in a smaller ${‖{\widehat{𝚯}}_0-{𝚯}_0^{*}‖}_F$. With an increase in the number of baseline samples, there is a improvement in the estimation of the baseline group network, leading to enhanced subgroup identification, clustering, and collaborative enhancement of network estimation for the heterogeneous group. In situations where the heterogeneous group exhibits an imbalanced mixture ratio, there is a certain degree of decline in the performance of evaluation measures. This is due to the challenges posed by the imbalanced characteristics to the parameter identification and estimation of the heterogeneous group, thereby affecting the estimation of the baseline group and causing simultaneous deterioration in both estimates. When the correct number of subgroups is assumed, SCAN and K-means methods demonstrate satisfactory advantages in clustering performance, reflected in lower CE. With an increase in the number of samples in baseline group, the advantages of these methods gradually diminish. Meanwhile, their clustering result does not exhibit ideal performance when the number of subgroups is incorrect.

Table 1.

Simulation 1 $(r=2,s=0.02,\mu =1.5)$ results under balanced and unbalanced settings: mean (standard deviation)

Setting	Method	Baseline group			Heterogeneous group
		${‖{\widehat{\mathit{\text{S}}}}_0-{\mathit{\text{S}}}_0^{*}‖}_F$	${‖{\widehat{\mathit{\text{L}}}}_0-{\mathit{\text{L}}}_0^{*}‖}_F$	${‖{\widehat{𝚯}}_0-{𝚯}_0^{*}‖}_F$	Per	$\widehat{\mathit{\text{K}}}$	CE	$\text{RMSE}\left(\right\{\mathit{\mu }\left\}\right)$	$\text{RMSE}\left(\right\{𝚫\left\}\right)$	$\text{TPR}\left(\right\{𝚫\left\}\right)$	$\text{FPR}\left(\right\{𝚫\left\}\right)$
$n_0=1500,{\pi }^{*}=(1/3,1/3,1/3)$	Proposed	1.650 (0.220)	0.965 (0.181)	1.661 (0.218)	0.930 (0.256)	2.940 (0.397)	0.025 (0.118)	0.525 (0.602)	4.540 (0.731)	0.720 (0.142)	0.015 (0.009)
	Separate	1.772 (0.354)	0.920 (0.190)	1.769 (0.330)	0.270 (0.446)	3.410 (0.753)	0.035 (0.080)	1.372 (0.822)	10.498 (1.896)	0.673 (0.111)	0.216 (0.049)
	$\text{SCAN}(K=6)$	–	–	–	0.000 (0.000)	6.000 (0.000)	0.166 (0.001)	2.502 (0.078)	5.993 (0.423)	0.746 (0.043)	0.194 (0.046)
	$\text{SCAN}(K=3)$	–	–	–	1.000 (0.000)	3.000 (0.000)	0.000 (0.000)	0.377 (0.037)	4.708 (0.371)	0.834 (0.037)	0.205 (0.046)
	$\text{K-means}(K=6)+\text{JGL}$	–	–	–	0.000 (0.000)	6.000 (0.000)	0.167 (0.001)	2.571 (0.075)	5.945 (0.410)	0.751 (0.042)	0.194 (0.046)
	$\text{K-means}(K=3)+\text{JGL}$	–	–	–	1.000 (0.000)	3.000 (0.000)	0.003 (0.012)	0.385 (0.074)	4.752 (0.399)	0.834 (0.036)	0.205 (0.046)
	$\text{K-means}(K=6)+\text{D-trace}$	–	–	–	0.000 (0.000)	6.000 (0.000)	0.167 (0.001)	2.690 (0.399)	5.687 (0.983)	0.682 (0.092)	0.020 (0.015)
	$\text{K-means}(K=3)+\text{D-trace}$	–	–	–	1.000 (0.000)	3.000 (0.000)	0.003 (0.012)	0.385 (0.074)	6.275 (1.103)	0.531 (0.136)	0.002 (0.003)
$n_0=1500,{\pi }^{*}=(1/6,1/3,1/2)$	Proposed	1.653 (0.229)	0.954 (0.174)	1.660 (0.221)	0.900 (0.302)	3.000 (0.471)	0.023 (0.106)	0.617 (0.662)	4.670 (0.752)	0.677 (0.137)	0.015 (0.010)
	Separate	1.772 (0.354)	0.920 (0.190)	1.769 (0.330)	0.290 (0.456)	2.690 (0.787)	0.057 (0.056)	1.171 (0.709)	8.900 (1.466)	0.726 (0.088)	0.225 (0.049)
	$\text{SCAN}(K=6)$	–	–	–	0.000 (0.000)	6.000 (0.000)	0.201 (0.013)	2.494 (0.104)	6.247 (0.453)	0.724 (0.047)	0.195 (0.046)
	$\text{SCAN}(K=3)$	–	–	–	1.000 (0.000)	3.000 (0.000)	0.000 (0.000)	0.421 (0.075)	4.957 (0.384)	0.817 (0.042)	0.208 (0.045)
	$\text{K-means}(K=6)+\text{JGL}$	–	–	–	0.000 (0.000)	6.000 (0.000)	0.202 (0.013)	2.560 (0.096)	6.195 (0.471)	0.731 (0.046)	0.196 (0.046)
	$\text{K-means}(K=3)+\text{JGL}$	–	–	–	1.000 (0.000)	3.000 (0.000)	0.013 (0.044)	0.507 (0.370)	5.046 (0.444)	0.813 (0.043)	0.208 (0.045)
	$\text{K-means}(K=6)+\text{D-trace}$	–	–	–	0.000 (0.000)	6.000 (0.000)	0.202 (0.014)	2.666 (0.384)	5.770 (0.959)	0.670 (0.089)	0.024 (0.018)
	$\text{K-means}(K=3)+\text{D-trace}$	–	–	–	1.000 (0.000)	3.000 (0.000)	0.018 (0.054)	0.556 (0.482)	4.563 (0.500)	0.799 (0.043)	0.148 (0.041)
$n_0=3000,{\pi }^{*}=(1/3,1/3,1/3)$	Proposed	0.987 (0.186)	0.765 (0.195)	0.953 (0.137)	0.930 (0.256)	3.000 (0.318)	0.011 (0.072)	0.514 (0.583)	4.314 (0.679)	0.719 (0.113)	0.014 (0.009)
	Separate	1.008 (0.224)	0.782 (0.197)	0.990 (0.167)	0.290 (0.456)	3.460 (0.731)	0.028 (0.074)	1.256 (0.634)	10.009 (1.910)	0.648 (0.101)	0.156 (0.046)
	$\text{SCAN}(K=6)$	–	–	–	0.000 (0.000)	6.000 (0.000)	0.166 (0.001)	2.498 (0.086)	5.529 (0.357)	0.721 (0.046)	0.132 (0.041)
	$\text{SCAN}(K=3)$	–	–	–	1.000 (0.000)	3.000 (0.000)	0.000 (0.000)	0.377 (0.032)	4.193 (0.280)	0.819 (0.036)	0.144 (0.041)
	$\text{K-means}(K=6)+\text{JGL}$	–	–	–	0.000 (0.000)	6.000 (0.000)	0.167 (0.001)	2.571 (0.082)	5.500 (0.352)	0.723 (0.043)	0.132 (0.041)
	$\text{K-means}(K=3)+\text{JGL}$	–	–	–	1.000 (0.000)	3.000 (0.000)	0.003 (0.013)	0.385 (0.072)	4.237 (0.340)	0.818 (0.037)	0.144 (0.041)
	$\text{K-means}(K=6)+\text{D-trace}$	–	–	–	0.000 (0.000)	6.000 (0.000)	0.167 (0.001)	2.676 (0.375)	5.652 (0.997)	0.694 (0.082)	0.024 (0.022)
	$\text{K-means}(K=3)+\text{D-trace}$	–	–	–	1.000 (0.000)	3.000 (0.000)	0.003 (0.013)	0.385 (0.072)	6.239 (1.115)	0.547 (0.134)	0.002 (0.004)
$n_0=3000,{\pi }^{*}=(1/6,1/3,1/2)$	Proposed	0.997 (0.207)	0.761 (0.203)	0.963 (0.169)	0.900 (0.302)	2.930 (0.498)	0.033 (0.135)	0.661 (0.796)	4.560 (0.856)	0.657 (0.176)	0.014 (0.009)
	Separate	1.008 (0.224)	0.782 (0.197)	0.990 (0.167)	0.260 (0.441)	2.840 (0.849)	0.049 (0.055)	1.170 (0.581)	8.739 (1.737)	0.700 (0.101)	0.165 (0.046)
	$\text{SCAN}(K=6)$	–	–	–	0.000 (0.000)	6.000 (0.000)	0.202 (0.013)	2.494 (0.114)	5.797 (0.430)	0.707 (0.047)	0.133 (0.041)
	$\text{SCAN}(K=3)$	–	–	–	1.000 (0.000)	3.000 (0.000)	0.002 (0.017)	0.452 (0.229)	4.456 (0.359)	0.804 (0.040)	0.148 (0.041)
	$\text{K-means}(K=6)+\text{JGL}$	–	–	–	0.000 (0.000)	6.000 (0.000)	0.203 (0.014)	2.557 (0.109)	5.739 (0.406)	0.711 (0.047)	0.134 (0.041)
	$\text{K-means}(K=3)+\text{JGL}$	–	–	–	1.000 (0.000)	3.000 (0.000)	0.018 (0.054)	0.556 (0.482)	4.563 (0.500)	0.799 (0.043)	0.148 (0.040)
	$\text{K-means}(K=6)+\text{D-trace}$	–	–	–	0.000 (0.000)	6.000 (0.000)	0.203 (0.014)	2.666 (0.376)	5.721 (0.963)	0.689 (0.100)	0.031 (0.028)
	$\text{K-means}(K=3)+\text{D-trace}$	–	–	–	1.000 (0.000)	3.000 (0.000)	0.018 (0.054)	0.556 (0.482)	6.282 (1.082)	0.542 (0.144)	0.003 (0.004)

In terms of differential precision matrix estimation, when the true number of subgroups is known, both SCAN(K = 3) and K-means(K = 3)+JGL demonstrate enhanced estimation performance due to their incorporation of accurate structural prior information (i.e., the true number of subgroups). Nevertheless, under these idealized conditions, our proposed method maintains a competitive edge, achieving the lowest RMSE values in most simulation in Table1. This relative advantage, however, diminishes progressively as network estimation complexity escalates, particularly with increased network density and confounding factors. This phenomenon highlight the importance of subgroup number in network learning. Notably, it is worth noting that in practical application scenarios, the actual number of subgroups is often unavailable, which makes it difficult to meet the preset conditions of SCAN(K = 3) and K-means(K = 3)+JGL. As shown from the results, the effectiveness of two step combinations of existing methods relies on prior information accuracy. For competing methods, except for the D-trace method, which is based on differential sparsity assumption, they model the precision matrix itself by imposing sparse penalties, generally showing higher FPR. Both the D-trace method and the proposed method are based on the differential sparsity assumption. Although the D-trace method has the lowest FPR when the true number of subgroups is given, the corresponding TPR is relatively low. In contrast, the proposed method maintains a moderate level of FPR while achieving higher TPR.

4. Application

In a global context, the incidence and mortality rates of cancer are gradually increasing, with lung cancer being the most common type. NSCLC is the leading cause of cancer-related mortality and accounts for 85% of lung cancers (Lahiri et al. 2023). In this section, we conduct a heterogeneous differential network analysis of NSCLC based on two public databases, TCGA and GTEx. We analyze the TcgaTargetGtex gene expected count dataset (Vivian et al. 2017) provided by UCSC Xena toilHub, where TCGA and GTEx samples are re-analyzed by the same RNA-seq pipeline. This dataset provides RNA-seq data for TCGA cancer samples from 16 primary sites and GTEx normal tissues. It is quantified using RNA-Seq by Expectation Maximization (RSEM) after unified read alignment, allowing for analysis of normal and cancer gene expression matrices under the same conditions. Because all samples are processed using a uniform bioinformatic pipeline, batch effects due to different computational processing is eliminated. We analyze a total of 797 TCGA NSCLC primary cancer samples, along with 288 GTEx normal lung tissue samples. The gene interactions within a biological network generally remain stable in normal lung tissue but exhibit significant variability in diseased tissues (Chen et al. 2021). Naturally, we take normal samples as the baseline group and NSCLC samples as the heterogeneous group. The gene set used in this study is derived from the Kyoto Encyclopedia of Genes and Genomes (KEGG)(Ogata et al. 1999) database, consisting of 72 genes playing important roles in NSCLC pathways.

When implementing our proposed approach, we set $K=6$ as the initial number of subgroups, which should be large enough. We adjust $r$ and tuning parameters through line search, with the algorithm step size chosen consistently with the strategy adopted in Xu et al. (2017) empirical analysis. Based on the AIC criterion in Formula (7), the line search selects rank of $r=6$. Following the implementation, we successfully identify three subgroup structures containing 462, 208, and 127 samples, respectively. Figure 3 visually displays the differential network structures corresponding to these three subgroups, with 273, 255, and 233 edges, respectively. The top 15 genes with the highest connectivity in these networks are marked in red. Additionally, Table 2 shows the overlap of edges in these three differential networks. The significant overlap of edges in these three differential networks reveals substantial commonalities among subgroups of NSCLC at the level of network variation.

Fig. 3.

Identified differential network structures corresponding to each subgroup (The top 15 genes with the highest connectivity are marked in red). Hub genes with connectivity in the differential network estimated by the method in this article (after standardization)

Table 2.

Overlap of identified differential networks

	Subgroup1	Subgroup2	Subgroup3
Subgroup1	273	208	194
Subgroup2	–	255	208
Subgroup3	–	–	233

Lung cancer is a highly heterogeneous disease with many forms of categorization. The main histological subtypes of NSCLC classified by the World Health Organization (WHO) are lung adenocarcinoma (LUAD) and lung squamous cell carcinoma (LUSC). This subtype label for each patient in our dataset is available. We compare our unsupervised heterogeneous analysis results with the histological classification obtained. Table 3 shows that our network-based subgroups are correlated with the classic subtypes. For example, LUSC subtypes are primarily distributed in the first group identified by our method, while LUAD subtypes are mainly in the second group. However, there are differences between the two grouping results. Our method more finely divides subgroups into three groups, with the third group containing a significant number of both LUAD and LUSC samples.

Table 3.

Comparison of our unsupervised heterogeneity analysis results with obtained histological classification

	Subgroup1	Subgroup2	Subgroup3	Sum
LUAD	64	185	71	320
LUSC	398	23	56	477
Sum	462	208	127	797

Considering the application of other competing methods, we follow the abbreviations of five alternative methods used in the simulation. The Separate method automatically select the number of subgroups automatically, and we also set $K=6$ as the initial subgroup number. For the comparison method that requires specifying the number of subgroups, we consider the case where $K$ is set to 2 (the number of histological classification) and $K=3$. For a fair comparison, we fix $r=6$ in each alternative method. The selection of other tuning parameters is obtained through line search. The number of subgroups estimated by these methods, clustering error(refer to histological classification), and the total number of edges in the differential network are reported in Table 4. In contrast, the subgroup analysis based on differential networks proposed in this article shows more reasonable results. Unlike sparse estimation of the precision matrix, our method and D-trace method are based on the assumption of differential sparsity. Therefore, an interpretable sparse differential network structure is obtained.

Table 4.

The clustering results of each method (refer to histological classification) and the total number of edges in the differential network estimation

Method	$\widehat{\mathit{K}}$	CE	Number
Proposed	3	0.302	761
Separate	5	0.426	7491
$\text{K-means}(K=2)$	–	0.471	–
$\text{K-means}(K=3)$	–	0.484	–
$\text{SCAN}(K=2)$	–	0.500	5008
$\text{SCAN}(K=3)$	–	0.438	7509
$\text{K-means}(K=2)+\text{JGL}$	–	–	5014
$\text{K-means}(K=3)+\text{JGL}$	–	–	7509
$\text{K-means}(K=2)+\text{D-trace}$	-	–	1279
$\text{K-means}(K=3)+\text{D-trace}$	–	–	874

A key challenge in differential network analysis is the absence of ground-truth networks for validation. Therefore, directly comparing the accuracy of different methods in estimating differential networks becomes difficult. To assess the performance of methods, we adopt another strategy: quantifying whether the set of hub nodes in differentially estimated networks can capture known functionally important genes. We hypothesize that methods performing better in capturing functionally important genes may also have superior performance in inferring differential networks. Based on our estimation results, we select the top 15 genes with the highest connectivity in each differential network as hub nodes. Figure 3 summarizes the hub nodes for three subgroups, displaying the number of connecting edges in their respective networks (normalized results, i.e., the number of edges connected by hub genes divided by the total number of edges in the network). Molecular-targeted therapy played an increasingly important role, particularly in genetically defined subgroups of patients. Among these hub nodes, EML4-ALK is a crucial fusion gene closely related to the occurrence and development of NSCLC. Targeted therapy against the EML4-ALK fusion gene is widely applied in personalized treatment for patients with LUAD (Lei et al. 2022). EGFR gene mutations have been extensively studied as biomarkers in clinical practice, serving as targets for cancer treatment (Fang et al. 2015). Ding et al. (2020) observe distinct patterns in the somatic mutation spectra between LUAD and LUSC. Compared to LUSC patients, LUAD patients exhibite more widespread oncogenic driver gene mutations, including EGFR deletion, ALK fusion, RET fusion, etc. Abnormal activation of EGFR in NSCLC can lead to the aberrant activation of multiple signaling pathways, including the RAS-MAPK pathway and PI3K-AKT-mTOR pathway, promoting the unlimited proliferation, survival, and invasion of cancer cells.

Our study demonstrate the potential of differential networks to explore cancer subtypes. We identify three network-based subgroups, revealing the substantial heterogeneity reflected in the differential networks of NSCLC patients. These subgroups are closely related to the histological classification. The philosophy of precision medicine advocates developing personalized treatment plans tailored to the unique characteristics of cancer. Therefore, it is important to focus on the unique patterns displayed in individual cancer samples to determine the most promising treatment strategy for patients. Our differential network subgroup analysis will promote the development of precision medicine. Although our experiment is conducted on NSCLC, the application of our method should not be limited to this single type of cancer. As long as there are corresponding samples as the baseline group, this method can be widely applied to subgroup and differential network analysis of any given heterogeneous samples. Therefore, it is a powerful resource that helps us to better understand the biological network changes that occur in individual patients.

5. Discussion

In this study, we develop a subgroup analysis of differential networks with latent variables under the framework of penalized likelihood, which characterizes the heterogeneity of differential networks between an unlabeled heterogeneous group and a baseline group. Simulation studies demonstrate satisfactory performance of network estimation and subgroup identification. The ability to estimate the baseline group network is enhanced, as the heterogeneous component provides auxiliary information for baseline group network in the joint estimation. Besides, the proposed method demonstrates relative advantages in clustering performance and differential network estimation. This strength benefits from the co-optimization strategy integrating subgroup division and network difference estimation. The fusion penalty mechanism enables data-driven determination of subgroup number, while also allowing users to specify the number of subgroups based on prior knowledge(by setting ${\lambda }_3=0$), demonstrating strong adaptability. Joint analysis of GTEx and TCGA data revealed that the proposed approach can provide a new view of disease heterogeneity, and the validity of the analysis has some indirect support. The proposed method is suitable for comparing network changes across different states (e.g., disease progression or pathological subtype differentiation). Beyond network analysis of omics data like gene expression, it is also applicable to other system-level investigations. For example, in neuroscience, it can identify connectivity differences between task-state and resting-state functional brain networks.

There are many directions for future research. Firstly, this paper primarily focuses on methodological and applied aspects, while theoretical foundations remain an underdeveloped area. Specifically, analyzing sparse plus low-rank structure under finite mixture of Gaussian graphical models demands the development of novel proof techniques, while the proposed joint modeling framework further complicates the theoretical analysis. Secondly, it is important to address the inherent assumption in our proposed method, which assumes independence between normal and cancer samples. To enhance the applicability of our method, future work should address paired sample analysis. For example, TCGA provides cancer tissue specimens and normal tissue specimens from the same patients. This involves explaining the interdependence between samples. Last but not least, our proposed method can be extended to accommodate non-normal data. An interesting work is to explore matrix variable differential network analysis using matrix normal mixture technology. The proposed method allows for analysis of multiple data types or patterns.

Appendix A. Additional simulation results

In the setting of $r=5,s=0.02,\mu =1.5$, we summarize the model performance of samples under balanced and unbalanced designs in Table 5. As the number of latent variables increases, the network estimation for the baseline group slightly decreases compared to the results when $r=2$. It is consistent with the findings of Xu et al. (2017). However, the proposed method shows an improvement in the proportion of identified subgroups for the heterogeneous group. Its advantage gradually becomes apparent in sample clustering. It is noteworthy that, in terms of the estimation of differential networks, the FPR of the other comparative methods, except for D-trace, significantly increases with the influence of an increased number of latent variables. The TPR of the D-trace method slightly decreases, whereas the proposed method remains stable without showing a pronounced change.

The model performance of the setting of $r=5,s=0.05,\mu =1.5$ is summarized in Table 6. We observe a declining trend in the performance of the methods represented by various metrics compared to the results when $s=0.02$ in Table 5, as the sparsity of the network increases. This can be explained by the decrease in network sparsity leading to an increase in the number of parameters to be estimated, thereby affecting the model’s performance under the same sample size. In terms of the proportion of identified subgroups, a trend contrary to the previous findings emerge, which we attribute to the randomness inherent in the simulation process. However, upon examining other metrics under this setting, our conclusions align with those in Table 1 of the main text.

Table 5.

Simulation 2 $(r=5,s=0.02,\mu =1.5)$ results under balanced and unbalanced settings: mean (standard deviation)

Setting	Method	Baseline group			Heterogeneous group
		${‖{\widehat{\mathit{\text{S}}}}_0-{\mathit{\text{S}}}_0^{*}‖}_F$	${‖{\widehat{\mathit{\text{L}}}}_0-{\mathit{\text{L}}}_0^{*}‖}_F$	${‖{\widehat{𝚯}}_0-{𝚯}_0^{*}‖}_F$	Per	$\widehat{\mathit{\text{K}}}$	CE	$\text{RMSE}\left(\right\{\mathit{\mu }\left\}\right)$	$\text{RMSE}\left(\right\{𝚫\left\}\right)$	$\text{TPR}\left(\right\{𝚫\left\}\right)$	$\text{FPR}\left(\right\{𝚫\left\}\right)$
$n_0=1500,{\pi }^{*}=(1/3,1/3,1/3)$	Proposed	1.710 (0.239)	1.461 (0.207)	1.772 (0.228)	0.960 (0.197)	2.990 (0.266)	0.009 (0.070)	0.440 (0.345)	4.429 (0.613)	0.727 (0.121)	0.017 (0.011)
	Separate	1.701 (0.215)	1.574 (0.190)	1.832 (0.222)	0.360 (0.482)	3.460 (0.658)	0.020 (0.062)	1.142 (0.637)	10.360 (1.794)	0.773 (0.080)	0.446 (0.054)
	$\text{SCAN}(K=6)$	–	–	–	0.000 (0.000)	6.000 (0.000)	0.166 (0.001)	2.486 (0.083)	5.860 (0.401)	0.821 (0.037)	0.432 (0.053)
	$\text{SCAN}(K=3)$	–	–	–	1.000 (0.000)	3.000 (0.000)	0.000 (0.000)	0.374 (0.032)	4.588 (0.346)	0.885 (0.028)	0.440 (0.052)
	$\text{K-means}(K=6)+\text{JGL}$	–	–	–	0.000 (0.000)	6.000 (0.000)	0.167 (0.001)	2.555 (0.079)	5.835 (0.394)	0.823 (0.035)	0.432 (0.053)
	$\text{K-means}(K=3)+\text{JGL}$	–	–	–	1.000 (0.000)	3.000 (0.000)	0.002 (0.005)	0.376 (0.040)	4.614 (0.348)	0.885 (0.028)	0.440 (0.052)
	$\text{K-means}(K=6)+\text{D-trace}$	–	–	–	0.000 (0.000)	6.000 (0.000)	0.167 (0.001)	2.621 (0.290)	5.693 (0.956)	0.692 (0.084)	0.023 (0.019)
	$\text{K-means}(K=3)+\text{D-trace}$	–	–	–	1.000 (0.000)	3.000 (0.000)	0.002 (0.005)	0.376 (0.040)	6.172 (0.996)	0.553 (0.114)	0.002 (0.003)
$n_0=1500,{\pi }^{*}=(1/6,1/3,1/2)$	Proposed	1.703 (0.220)	1.451 (0.206)	1.758 (0.219)	0.910 (0.288)	2.970 (0.388)	0.017 (0.088)	0.549 (0.531)	4.580 (0.668)	0.700 (0.121)	0.019 (0.012)
	Separate	1.701 (0.215)	1.574 (0.190)	1.832 (0.222)	0.220 (0.416)	2.680 (0.827)	0.061 (0.056)	1.193 (0.570)	8.784 (1.740)	0.809 (0.075)	0.455 (0.056)
	$\text{SCAN}(K=6)$	–	–	–	0.000 (0.000)	6.000 (0.000)	0.206 (0.016)	2.461 (0.118)	6.069 (0.411)	0.805 (0.042)	0.432 (0.053)
	$\text{SCAN}(K=3)$	–	–	–	1.000 (0.000)	3.000 (0.000)	0.002 (0.024)	0.435 (0.248)	4.845 (0.338)	0.871 (0.033)	0.441 (0.052)
	$\text{K-means}(K=6)+\text{JGL}$	–	–	–	0.000 (0.000)	6.000 (0.000)	0.207 (0.016)	2.526 (0.113)	6.012 (0.408)	0.812 (0.040)	0.433 (0.053)
	$\text{K-means}(K=3)+\text{JGL}$	–	–	–	1.000 (0.000)	3.000 (0.000)	0.006 (0.031)	0.454 (0.295)	4.889 (0.362)	0.870 (0.034)	0.441 (0.052)
	$\text{K-means}(K=6)+\text{D-trace}$	–	–	–	0.000 (0.000)	6.000 (0.000)	0.207 (0.016)	2.596 (0.328)	5.784 (0.964)	0.665 (0.104)	0.024 (0.022)
	$\text{K-means}(K=3)+\text{D-trace}$	–	–	–	1.000 (0.000)	3.000 (0.000)	0.006 (0.031)	0.454 (0.295)	6.189 (1.002)	0.544 (0.120)	0.003 (0.005)
$n_0=3000,{\pi }^{*}=(1/3,1/3,1/3)$	Proposed	1.108 (0.224)	1.161 (0.229)	1.070 (0.162)	0.950 (0.219)	3.000 (0.284)	0.009 (0.070)	0.481 (0.506)	4.199 (0.521)	0.734 (0.094)	0.015 (0.008)
	Separate	1.106 (0.220)	1.252 (0.210)	1.150 (0.160)	0.300 (0.461)	3.420 (0.727)	0.031 (0.076)	1.231 (0.620)	9.836 (1.893)	0.724 (0.078)	0.324 (0.055)
	$\text{SCAN}(K=6)$	–	–	–	0.000 (0.000)	6.000 (0.000)	0.166 (0.001)	2.493 (0.086)	5.457 (0.360)	0.777 (0.036)	0.305 (0.053)
	$\text{SCAN}(K=3)$	–	–	–	1.000 (0.000)	3.000 (0.000)	0.000 (0.000)	0.381 (0.034)	4.128 (0.310)	0.857 (0.033)	0.314 (0.053)
	$\text{K-means}(K=6)+\text{JGL}$	–	–	–	0.000 (0.000)	6.000 (0.000)	0.167 (0.001)	2.563 (0.081)	5.419 (0.387)	0.781 (0.036)	0.305 (0.053)
	$\text{K-means}(K=3)+\text{JGL}$	–	–	–	1.000 (0.000)	3.000 (0.000)	0.002 (0.005)	0.384 (0.040)	4.153 (0.317)	0.856 (0.032)	0.314 (0.052)
	$\text{K-means}(K=6)+\text{D-trace}$	–	–	–	0.000 (0.000)	6.000 (0.000)	0.167 (0.001)	2.635 (0.298)	5.676 (0.973)	0.707 (0.094)	0.031 (0.035)
	$\text{K-means}(K=3)+\text{D-trace}$	–	–	–	1.000 (0.000)	3.000 (0.000)	0.002 (0.005)	0.384 (0.040)	6.193 (0.994)	0.544 (0.121)	0.001 (0.002)
$n_0=3000,{\pi }^{*}=(1/6,1/3,1/2)$	Proposed	1.122 (0.252)	1.164 (0.234)	1.086 (0.194)	0.950 (0.219)	2.970 (0.332)	0.015 (0.087)	0.532 (0.492)	4.399 (0.645)	0.693 (0.124)	0.015 (0.009)
	Separate	1.106 (0.220)	1.252 (0.210)	1.150 (0.160)	0.290 (0.456)	2.710 (0.795)	0.055 (0.056)	1.128 (0.591)	8.279 (1.867)	0.767 (0.084)	0.332 (0.057)
	$\text{SCAN}(K=6)$	–	–	–	0.000 (0.000)	6.000 (0.000)	0.204 (0.015)	2.474 (0.122)	5.681 (0.399)	0.763 (0.041)	0.305 (0.053)
	$\text{SCAN}(K=3)$	–	–	–	1.000 (0.000)	3.000 (0.000)	0.002 (0.023)	0.456 (0.254)	4.411 (0.332)	0.840 (0.039)	0.316 (0.053)
	$\text{K-means}(K=6)+\text{JGL}$	–	–	–	0.000 (0.000)	6.000 (0.000)	0.206 (0.016)	2.541 (0.115)	5.639 (0.393)	0.768 (0.039)	0.305 (0.053)
	$\text{K-means}(K=3)+\text{JGL}$	–	–	–	1.000 (0.000)	3.000 (0.000)	0.017 (0.052)	0.559 (0.487)	4.512 (0.405)	0.837 (0.039)	0.316 (0.053)
	$\text{K-means}(K=6)+\text{D-trace}$	–	–	–	0.000 (0.000)	6.000 (0.000)	0.206 (0.016)	2.617 (0.338)	5.755 (0.978)	0.690 (0.097)	0.035 (0.038)
	$\text{K-means}(K=3)+\text{D-trace}$	–	–	–	1.000 (0.000)	3.000 (0.000)	0.017 (0.052)	0.559 (0.488)	6.256 (0.995)	0.533 (0.122)	0.002 (0.003)

The model performance of the setting of $r=5,s=0.05,\mu =2$ is summarized in Table 7. Compared to the results in Table 6, the ability to correctly identify the number of subgroups is improved by increasing the distance between subgroups. Additionally, most measures representing model performance show an upward trend.

Appendix B. Details of Algorithm

Minimizing (4) with respect to $(\mathit{\mu },𝚫,\mathit{\pi })$ yields the update of parameters. We have

$${\pi }_k^{\left(m\right)}=\frac{1}{n_1}\sum _{i:y_i=1}{\gamma }_{ik}^{\left(m\right)}.$$

Minimizing (4) with respect to $\mathit{\mu }$ is equivalent to solving

$$\begin{gathered} {\mathit{\mu }}^{\left(m\right)}=\underset{\mathit{\mu }}{\text{argmin}}\frac{1}{2n_1}\sum _{i:y_i=1}\sum _{k=1}^K{\gamma }_{ik}^{\left(m\right)} \\ \times \left\{{\left({\mathit{x}}_i-{\mathit{\mu }}_k\right)}^{\top }{𝚯}_k^{(m-1)}\left({\mathit{x}}_i-{\mathit{\mu }}_k\right)\right\}+𝒫\left(\mathit{\mu }\right). \end{gathered}$$

For this problem, we adopt the local quadratic approximation:

$$\begin{gathered} {\mathit{\mu }}^{\left(m\right)}=\underset{\mathit{\mu }}{\text{argmin}}\frac{1}{2n_1}\sum _{i:y_i=1}\sum _{k=1}^K{\gamma }_{ik}^{\left(m\right)} \\ \times \left({\left({\mathit{x}}_i-{\mathit{\mu }}_k\right)}^{\top }{𝚯}_k^{(m-1)}\left({\mathit{x}}_i-{\mathit{\mu }}_k\right)\right) \\ +\sum _{k<k^{\prime }}\frac{1}{2}\frac{p^{\prime }\left({\tau }_{k,k^{\prime }}^{(m-1,m-1)},{\lambda }_3\right)}{{\tau }_{k,k^{\prime }}^{(m-1,m-1)}}{‖{\mathit{\mu }}_k-{\mathit{\mu }}_{k^{\prime }}‖}_2^2, \end{gathered}$$

which leads to an explicit solution of ${\mathit{\mu }}_k^{\left(m\right)}$ as follows: For $j=1,\dots ,p$,

$${\mu }_{kj}^{\left(m\right)}=\frac{h_j^{(m-1)}+n_1{\tilde{v}}_k^{(m-1)}}{n_1{\widehat{v}}_k^{(m-1)}+n_{1k}{\theta }_{kjj}^{(m-1)}},$$

where $n_{1k}=\sum _{i:y_i=1}{\gamma }_{ik}$,

$$h_j^{\left(m-1\right)}=\sum _{i:y_i=1}{\gamma }_{ik}^{\left(m\right)}\left(\sum _{l\ne j}^p{\theta }_{kjl}^{\left(m-1\right)}\left(x_{il}-{\mu }_{kl}^{\left(m-1\right)}I(l>j)\left.-{\mu }_{kl}^{\left(m\right)}I(l<j)\right)+{\theta }_{kjj}^{\left(m-1\right)}{x}_{ij}\right.\right)$$

$$\begin{gathered} {\tilde{v}}_k^{\left(m-1\right)}=\sum _{1⩽k^{\prime }<k}\frac{p^{\prime }\left({\tilde{\tau }}_{k,k^{\prime }}^{\left(m-1\right)},{\lambda }_3\right)}{{\tilde{\tau }}_{k,k^{\prime }}^{\left(m-1\right)}}{\mu }_{k^{\prime }j}^{\left(m\right)} \\ +\sum _{k<k^{\prime }⩽K}\frac{p^{\prime }\left({\tilde{\tau }}_{k,k^{\prime }}^{\left(m-1\right)},{\lambda }_3\right)}{{\tilde{\tau }}_{k,k^{\prime }}^{\left(m-1\right)}}{\mu }_{k^{\prime }j}^{\left(m-1\right)}, \end{gathered}$$

$${\widehat{v}}_k^{\left(m-1\right)}=\sum _{k\ne k^{\prime }}\frac{p^{\prime }\left({\tilde{\tau }}_{k,k^{\prime }}^{\left(m\right)},{\lambda }_3\right)}{{\tilde{\tau }}_{k,k^{\prime }}^{\left(m-1\right)}},$$

$$\begin{gathered} {\tilde{\tau }}_{k,k^{\prime }}^{\left(m-1\right)}={\tau }_{k,k^{\prime }}^{\left(m-1,m\right)}I\left(1⩽k^{\prime }<k\right) \\ +{\tau }_{k,k^{\prime }}^{\left(m-1,m-1\right)}I\left(k<k^{\prime }⩽K\right) \end{gathered}$$

$${\tau }_{k,k^{\prime }}^{\left(r_1,r_2\right)}={\left({‖{\mathit{\mu }}_k^{\left(r_1\right)}-{\mathit{\mu }}_{k^{\prime }}^{\left(r_2\right)}‖}_2^2+{‖{\mathit{\Theta }}_k^{\left(m-1\right)}-{\mathit{\Theta }}_{k^{\prime }}^{\left(m-1\right)}‖}_F^2\right)}^{1/2},$$

and ${𝚯}_k^{(m-1)},{𝚯}_{k^{\prime }}^{(m-1)}$ are estimators from the $(m-1)$-th iteration.

Minimizing (4) with respect to $\left\{{\mathbf{\Theta }}_k,k=1,\dots ,K\right\}$ is equivalent to solving

$${𝚯}^{\left(m\right)}=\underset{𝚯}{\text{argmin}}(-\sum _{k=1}^K{n}_{1k}\left(\text{log}\text{det}\left({𝚯}_k\right)-\text{tr}\left({\widehat{𝚺}}_k{\Theta }_k\right)\right)+𝒫(𝚯)).$$

This can be efficiently achieved using the ADMM algorithm by modifying the JGL algorithm in Danaher et al. (2014). More specifically, this optimization can be reformulated as

$${𝚯}^{\left(m\right)}=\underset{𝚯,𝚵}{\text{argmin}}\left(-\sum _{k=1}^K{n}_{1k}\left(\text{log}\text{det}\left({𝚯}_k\right)\left.-\text{tr}\left({\widehat{𝚺}}_k{𝚯}_k\right)\right)+𝒫(𝚵)\right.\right),$$ (B1)

$$s.t.{𝚵}_k={𝚯}_k-{𝚯}_0,{𝚯}_k\in {𝒮}_{++}^p,k=1,\dots ,K,$$

where ${𝚯}_k\in {𝒮}_{++}^p,k=1,\dots ,K$ are the positive-definite constraints, and $𝚵=\left\{{𝚵}_1,\dots ,{𝚵}_K\right\},{𝚵}_k={\left({\xi }_{kij}\right)}_{1⩽i,j⩽p}$. The scaled augmented Lagrangian form for this problem is given by

$$\begin{gathered} {𝒬}_{\kappa }(𝚯,𝚵,𝚿)=-\sum _{k=1}^K{n}_{1k}\left(\text{log}\text{det}\left({𝚯}_k\right)\right. \\ \left.-\text{tr}\left({\widehat{𝚺}}_k{𝚯}_k\right)\right)+𝒫\left(𝚵\right) \\ +\frac{\kappa }{2}\sum _{k=1}^K{‖{𝚯}_k-{𝚯}_0-{𝚵}_k+{𝚿}_k‖}_{\text{F}}^2 \\ -\frac{\kappa }{2}\sum _{k=1}^K{‖{𝚿}_k‖}_{\text{F}}^2, \end{gathered}$$

where $𝚿=\left\{{𝚿}_1,\dots ,{𝚿}_K\right\}$ are dual variables, and $\kappa$ is the penalty parameter. Throughout this study, we set $\kappa =1$. In the $r$–step of the ADMM algorithm for solving (B1), the iterative procedure can be summarized as follows:

Step1: For $k=1,\dots ,K$, update ${𝚯}_k^{\left(r\right)}$ by solving

$$\underset{𝚯}{\text{argmin}}\left(-n_k\left(\text{log}\text{det}\left({𝚯}_k\right)-\text{tr}\left({\widehat{𝚺}}_k{𝚯}_k\right)\right)\right.\left.+\frac{\kappa }{2}{‖{𝚯}_k-{𝚯}_0-{𝚵}_k^{(r-1)}+{𝚿}_k^{(r-1)}‖}_{\text{F}}^2\right).$$

Table 6.

Simulation 3 $(r=5,s=0.05,\mu =1.5)$ results under balanced and unbalanced settings: mean (standard deviation)

Setting	Method	Baseline group			Heterogeneous group
		${‖{\widehat{\mathit{\text{S}}}}_0-{\mathit{\text{S}}}_0^{*}‖}_F$	${‖{\widehat{\mathit{\text{L}}}}_0-{\mathit{\text{L}}}_0^{*}‖}_F$	${‖{\widehat{𝚯}}_0-{𝚯}_0^{*}‖}_F$	Per	$\widehat{\mathit{\text{K}}}$	CE	$\text{RMSE}\left(\right\{\mathit{\mu }\left\}\right)$	$\text{RMSE}\left(\right\{𝚫\left\}\right)$	$\text{TPR}\left(\right\{𝚫\left\}\right)$	$\text{FPR}\left(\right\{𝚫\left\}\right)$
$n_0=1500,{\pi }^{*}=(1/3,1/3,1/3)$	Proposed	2.607 (0.289)	1.883 (0.228)	2.702 (0.259)	0.850 (0.359)	3.060 (0.547)	0.024 (0.116)	0.633 (0.815)	8.210 (0.818)	0.647 (0.137)	0.036 (0.013)
	Separate	2.645 (0.332)	1.878 (0.236)	2.732 (0.284)	0.100 (0.302)	2.390 (0.827)	0.181 (0.148)	1.849 (0.981)	16.065 (2.337)	0.829 (0.071)	0.589 (0.062)
	$\text{SCAN}(K=6)$	–	–	–	0.000 (0.000)	6.000 (0.000)	0.166 (0.001)	2.529 (0.049)	10.086 (0.525)	0.830 (0.027)	0.518 (0.046)
	$\text{SCAN}(K=3)$	–	–	–	1.000 (0.000)	3.000 (0.000)	0.000 (0.000)	0.333 (0.030)	7.766 (0.475)	0.891 (0.020)	0.532 (0.045)
	$\text{K-means}(K=6)+\text{JGL}$	–	–	–	0.000 (0.000)	6.000 (0.000)	0.166 (0.001)	2.562 (0.046)	10.096 (0.522)	0.832 (0.027)	0.517 (0.047)
	$\text{K-means}(K=3)+\text{JGL}$	–	–	–	1.000 (0.000)	3.000 (0.000)	0.001 (0.001)	0.333 (0.029)	7.788 (0.493)	0.890 (0.020)	0.532 (0.045)
	$\text{K-means}(K=6)+\text{Dtrace}$	–	–	–	0.000 (0.000)	6.000 (0.000)	0.166 (0.001)	2.733 (0.477)	9.608 (0.874)	0.568 (0.120)	0.029 (0.024)
	$\text{K-means}(K=3)+\text{Dtrace}$	–	–	–	1.000 (0.000)	3.000 (0.000)	0.001 (0.001)	0.333 (0.029)	10.097 (1.069)	0.450 (0.149)	0.006 (0.008)
$n_0=1500,{\pi }^{*}=(1/6,1/3,1/2)$	Proposed	2.618 (0.288)	1.876 (0.234)	2.709 (0.254)	0.870 (0.338)	2.970 (0.502)	0.027 (0.121)	0.658 (0.887)	8.386 (0.795)	0.595 (0.142)	0.034 (0.013)
	Separate	2.645 (0.332)	1.878 (0.236)	2.732 (0.284)	0.050 (0.219)	2.040 (0.425)	0.129 (0.115)	1.584 (0.970)	14.825 (1.699)	0.859 (0.060)	0.599 (0.061)
	$\text{SCAN}(K=6)$	–	–	–	0.000 (0.000)	6.000 (0.000)	0.201 (0.013)	2.522 (0.082)	10.482 (0.559)	0.816 (0.032)	0.517 (0.046)
	$\text{SCAN}(K=3)$	–	–	–	1.000 (0.000)	3.000 (0.000)	0.000 (0.000)	0.367 (0.046)	8.201 (0.449)	0.881 (0.020)	0.532 (0.046)
	$\text{K-means}(K=6)+\text{JGL}$	–	–	–	0.000 (0.000)	6.000 (0.000)	0.201 (0.013)	2.553 (0.079)	10.479 (0.544)	0.818 (0.032)	0.517 (0.047)
	$\text{K-means}(K=3)+\text{JGL}$	–	–	–	1.000 (0.000)	3.000 (0.000)	0.002 (0.016)	0.378 (0.134)	8.240 (0.481)	0.881 (0.021)	0.532 (0.046)
	$\text{K-means}(K=6)+\text{Dtrace}$	–	–	–	0.000 (0.000)	6.000 (0.000)	0.201 (0.013)	2.728 (0.511)	9.671 (0.844)	0.544 (0.123)	0.030 (0.023)
	$\text{K-means}(K=3)+\text{Dtrace}$	–	–	–	1.000 (0.000)	3.000 (0.000)	0.002 (0.016)	0.378 (0.134)	10.148 (1.100)	0.432 (0.155)	0.006 (0.007)
$n_0=3000,{\pi }^{*}=(1/3,1/3,1/3)$	Proposed	1.706 (0.257)	1.510 (0.237)	1.667 (0.167)	0.830 (0.378)	3.170 (0.378)	0.001 (0.001)	0.580 (0.543)	8.022 (0.778)	0.646 (0.095)	0.032 (0.011)
	Separate	1.727 (0.290)	1.536 (0.228)	1.709 (0.182)	0.120 (0.327)	2.300 (0.798)	0.199 (0.167)	1.863 (1.084)	15.004 (2.230)	0.799 (0.084)	0.487 (0.078)
	$\text{SCAN}(K=6)$	–	–	–	0.000 (0.000)	6.000 (0.000)	0.166 (0.001)	2.535 (0.045)	9.449 (0.477)	0.786 (0.029)	0.387 (0.058)
	$\text{SCAN}(K=3)$	–	–	–	1.000 (0.000)	3.000 (0.000)	0.000 (0.000)	0.339 (0.031)	7.099 (0.452)	0.862 (0.021)	0.405 (0.056)
	$\text{K-means}(K=6)+\text{JGL}$	–	–	–	0.000 (0.000)	6.000 (0.000)	0.166 (0.001)	2.569 (0.043)	9.405 (0.493)	0.790 (0.029)	0.388 (0.058)
	$\text{K-means}(K=3)+\text{JGL}$	–	–	–	1.000 (0.000)	3.000 (0.000)	0.000 (0.001)	0.339 (0.031)	7.119 (0.452)	0.861 (0.022)	0.404 (0.056)
	$\text{K-means}(K=6)+\text{Dtrace}$	–	–	–	0.000 (0.000)	6.000 (0.000)	0.166 (0.001)	2.727 (0.465)	9.508 (0.873)	0.593 (0.122)	0.036 (0.033)
	$\text{K-means}(K=3)+\text{Dtrace}$	–	–	–	1.000 (0.000)	3.000 (0.000)	0.000 (0.001)	0.339 (0.031)	10.095 (1.043)	0.454 (0.145)	0.005 (0.005)
$n_0=3000,{\pi }^{*}=(1/6,1/3,1/2)$	Proposed	1.722 (0.320)	1.511 (0.269)	1.691 (0.255)	0.880 (0.327)	3.010 (0.460)	0.020 (0.104)	0.622 (0.727)	8.231 (0.863)	0.590 (0.138)	0.030 (0.013)
	Separate	1.727 (0.290)	1.536 (0.228)	1.709 (0.182)	0.060 (0.239)	2.090 (0.452)	0.116 (0.095)	1.602 (0.954)	14.261 (1.506)	0.814 (0.084)	0.485 (0.067)
	$\text{SCAN}(K=6)$	–	–	–	0.000 (0.000)	6.000 (0.000)	0.201 (0.014)	2.526 (0.092)	9.858 (0.477)	0.770 (0.039)	0.387 (0.059)
	$\text{SCAN}(K=3)$	–	–	–	1.000 (0.000)	3.000 (0.000)	0.002 (0.015)	0.399 (0.226)	7.606 (0.564)	0.846 (0.032)	0.404 (0.060)
	$\text{K-means}(K=6)+\text{JGL}$	–	–	–	0.000 (0.000)	6.000 (0.000)	0.201 (0.014)	2.558 (0.087)	9.829 (0.491)	0.773 (0.040)	0.387 (0.060)
	$\text{K-means}(K=3)+\text{JGL}$	–	–	–	1.000 (0.000)	3.000 (0.000)	0.007 (0.036)	0.431 (0.350)	7.637 (0.589)	0.845 (0.030)	0.405 (0.060)
	$\text{K-means}(K=6)+\text{Dtrace}$	–	–	–	0.000 (0.000)	6.000 (0.000)	0.201 (0.014)	2.711 (0.466)	9.532 (0.881)	0.581 (0.126)	0.042 (0.036)
	$\text{K-means}(K=3)+\text{Dtrace}$	–	–	–	1.000 (0.000)	3.000 (0.000)	0.007 (0.036)	0.431 (0.351)	10.117 (1.022)	0.451 (0.147)	0.007 (0.010)

Table 7.

Simulation 4 ($r=5,s=0.05,\mu =2)$ results under balanced and unbalanced settings: mean (standard deviation)

Setting	Method	Baseline group			Heterogeneous group
		${‖{\widehat{\mathit{\text{S}}}}_0-{\mathit{\text{S}}}_0^{*}‖}_F$	${‖{\widehat{\mathit{\text{L}}}}_0-{\mathit{\text{L}}}_0^{*}‖}_F$	${‖{\widehat{𝚯}}_0-{𝚯}_0^{*}‖}_F$	Per	$\widehat{\mathit{\text{K}}}$	CE	$\text{RMSE}\left(\right\{\mathit{\mu }\left\}\right)$	$\text{RMSE}\left(\right\{𝚫\left\}\right)$	$\text{TPR}\left(\right\{𝚫\left\}\right)$	$\text{FPR}\left(\right\{𝚫\left\}\right)$
$n_0=1500,{\pi }^{*}=(1/3,1/3,1/3)$	Proposed	2.596 (0.277)	1.880 (0.225)	2.688 (0.249)	0.950 (0.219)	2.970 (0.413)	0.021 (0.115)	0.515 (0.895)	8.130 (0.765)	0.659 (0.135)	0.037 (0.013)
	Separate	2.645 (0.332)	1.878 (0.236)	2.732 (0.284)	0.200 (0.402)	2.510 (0.772)	0.144 (0.117)	1.889 (1.000)	16.345 (2.096)	0.843 (0.060)	0.583 (0.056)
	$\text{SCAN}(K=6)$	–	–	–	0.000 (0.000)	6.000 (0.000)	0.166 (0.001)	2.528 (0.049)	10.066 (0.509)	0.831 (0.027)	0.518 (0.046)
	$\text{SCAN}(K=3)$	–	–	–	1.000 (0.000)	3.000 (0.000)	0.000 (0.000)	0.332 (0.029)	7.766 (0.475)	0.891 (0.020)	0.532 (0.045)
	$\text{K-means}(K=6)+\text{JGL}$	–	–	–	0.000 (0.000)	6.000 (0.000)	0.166 (0.001)	2.561 (0.046)	10.104 (0.515)	0.831 (0.027)	0.517 (0.047)
	$\text{K-means}(K=3)+\text{JGL}$	–	–	–	1.000 (0.000)	3.000 (0.000)	0.000 (0.000)	0.332 (0.029)	7.766 (0.475)	0.891 (0.020)	0.532 (0.045)
	$\text{K-means}(K=6)+\text{D-trace}$	–	–	–	0.000 (0.000)	6.000 (0.000)	0.166 (0.001)	2.620 (0.306)	9.686 (0.885)	0.582 (0.103)	0.029 (0.024)
	$\text{K-means}(K=3)+\text{D-trace}$	–	–	–	1.000 (0.000)	3.000 (0.000)	0.000 (0.000)	0.332 (0.029)	10.108 (1.064)	0.446 (0.148)	0.005 (0.007)
$n_0=1500,{\pi }^{*}=(1/6,1/3,1/2)$	Proposed	2.628 (0.352)	1.868 (0.231)	2.714 (0.316)	0.950 (0.219)	3.020 (0.284)	0.007 (0.062)	0.471 (0.564)	8.291 (0.723)	0.628 (0.091)	0.037 (0.011)
	Separate	2.645 (0.332)	1.878 (0.236)	2.732 (0.284)	0.070 (0.256)	2.190 (0.581)	0.106 (0.083)	1.784 (0.888)	15.337 (1.952)	0.859 (0.061)	0.595 (0.053)
	$\text{SCAN}(K=6)$	–	–	–	0.000 (0.000)	6.000 (0.000)	0.201 (0.013)	2.522 (0.082)	10.482 (0.559)	0.816 (0.033)	0.517 (0.047)
	$\text{SCAN}(K=3)$	–	–	–	1.000 (0.000)	3.000 (0.000)	0.000 (0.000)	0.363 (0.028)	8.201 (0.449)	0.881 (0.020)	0.532 (0.046)
	$\text{K-means}(K=6)+\text{JGL}$	–	–	–	0.000 (0.000)	6.000 (0.000)	0.201 (0.013)	2.551 (0.079)	10.446 (0.562)	0.819 (0.033)	0.518 (0.047)
	$\text{K-means}(K=3)+\text{JGL}$	–	–	–	1.000 (0.000)	3.000 (0.000)	0.000 (0.000)	0.363 (0.028)	8.201 (0.449)	0.881 (0.020)	0.532 (0.046)
	$\text{K-means}(K=6)+\text{D-trace}$	–	–	–	0.000 (0.000)	6.000 (0.000)	0.201 (0.013)	2.617 (0.372)	9.744 (0.841)	0.558 (0.109)	0.030 (0.023)
	$\text{K-means}(K=3)+\text{D-trace}$	–	–	–	1.000 (0.000)	3.000 (0.000)	0.000 (0.000)	0.363 (0.028)	10.159 (1.103)	0.430 (0.152)	0.006 (0.007)
$n_0=3000,{\pi }^{*}=(1/3,1/3,1/3)$	Proposed	1.706 (0.260)	1.527 (0.245)	1.683 (0.189)	0.910 (0.288)	3.060 (0.343)	0.007 (0.067)	0.483 (0.497)	7.985 (0.756)	0.654 (0.117)	0.035 (0.026)
	Separate	1.727 (0.290)	1.536 (0.228)	1.709 (0.182)	0.190 (0.394)	2.490 (0.785)	0.150 (0.127)	1.957 (1.023)	15.571 (2.094)	0.807 (0.079)	0.479 (0.075)
	$\text{SCAN}(K=6)$	–	–	–	0.000 (0.000)	6.000 (0.000)	0.166 (0.001)	2.535 (0.045)	9.448 (0.467)	0.786 (0.029)	0.387 (0.058)
	$\text{SCAN}(K=3)$	–	–	–	1.000 (0.000)	3.000 (0.000)	0.000 (0.000)	0.339 (0.030)	7.099 (0.452)	0.862 (0.021)	0.405 (0.056)
	$\text{K-means}(K=6)+\text{JGL}$	–	–	–	0.000 (0.000)	6.000 (0.000)	0.166 (0.001)	2.568 (0.043)	9.387 (0.491)	0.791 (0.029)	0.388 (0.058)
	$\text{K-means}(K=3)+\text{JGL}$	–	–	–	1.000 (0.000)	3.000 (0.000)	0.000 (0.000)	0.339 (0.030)	7.099 (0.452)	0.862 (0.021)	0.405 (0.056)
	$\text{K-means}(K=6)+\text{Dtrace}$	–	–	–	0.000 (0.000)	6.000 (0.000)	0.166 (0.001)	2.637 (0.351)	9.569 (0.890)	0.605 (0.107)	0.036 (0.034)
	$\text{K-means}(K=3)+\text{Dtrace}$	–	–	–	1.000 (0.000)	3.000 (0.000)	0.000 (0.000)	0.339 (0.030)	10.098 (1.043)	0.453 (0.143)	0.005 (0.005)
$n_0=3000,{\pi }^{*}=(1/6,1/3,1/2)$	Proposed	1.711 (0.264)	1.520 (0.240)	1.678 (0.193)	0.940 (0.239)	2.990 (0.301)	0.008 (0.063)	0.478 (0.517)	8.118 (0.644)	0.624 (0.093)	0.033 (0.013)
	Separate	1.727 (0.290)	1.536 (0.228)	1.709 (0.182)	0.100 (0.302)	2.140 (0.472)	0.107 (0.082)	1.726 (0.902)	14.515 (1.513)	0.829 (0.078)	0.492 (0.074)
	$\text{SCAN}(K=6)$	–	–	–	0.000 (0.000)	6.000 (0.000)	0.201 (0.014)	2.525 (0.090)	9.850 (0.474)	0.770 (0.040)	0.387 (0.059)
	$\text{SCAN}(K=3)$	–	–	–	1.000 (0.000)	3.000 (0.000)	0.000 (0.000)	0.370 (0.034)	7.575 (0.424)	0.847 (0.027)	0.404 (0.060)
	$\text{K-means}(K=6)+\text{JGL}$	–	–	–	0.000 (0.000)	6.000 (0.000)	0.201 (0.013)	2.557 (0.087)	9.819 (0.509)	0.774 (0.039)	0.387 (0.060)
	$\text{K-means}(K=3)+\text{JGL}$	–	–	–	1.000 (0.000)	3.000 (0.000)	0.000 (0.000)	0.370 (0.034)	7.575 (0.424)	0.847 (0.027)	0.404 (0.060)
	$\text{K-means}(K=6)+\text{D-trace}$	–	–	–	0.000 (0.000)	6.000 (0.000)	0.201 (0.013)	2.596 (0.296)	9.607 (0.913)	0.595 (0.111)	0.042 (0.037)
	$\text{K-means}(K=3)+\text{D-trace}$	–	–	–	1.000 (0.000)	3.000 (0.000)	0.000 (0.000)	0.370 (0.034)	10.125 (1.030)	0.448 (0.147)	0.006 (0.008)

Following Witten and Tibshirani (2009), the solution is given by

$${𝚯}_k^{\left(r\right)}=\mathit{U}\tilde{\mathit{D}}{\mathit{U}}^{\top },$$

where $\mathit{U}\mathit{D}{\mathit{U}}^{\top }$ is the eigendecomposition of ${\widehat{𝚺}}_k-\kappa {\Xi }_k^{(r-1)}/n_{1k}-\kappa {𝚯}_0/n_{1k}+\kappa {𝚿}_k^{(r-1)}/n_{1k},\tilde{\mathit{D}}$ is a diagonal matrix with the $j$ th diagonal element $\frac{n_{1k}}{2\kappa }\left(-{\mathit{D}}_{jj}+{\left({\mathit{D}}_{jj}^2+4\kappa /n_{1k}\right)}^{1/2}\right)$.

Step2: For $k=1,\dots ,K$, update ${𝚵}_k^{\left(r\right)}$ by solving

$$\underset{𝚵}{\text{argmin}}\left(\frac{\kappa }{2}\sum _{k=1}^K{‖{𝚵}_k-{\mathcal{Z}}_k‖}_{\text{F}}^2+𝒫(𝚵)\right),$$ (B2)

where ${\mathcal{Z}}_k={𝚯}_k^{\left(r\right)}-{𝚯}_0+{𝚿}_k^{(r-1)}$. The implementation uses the sparse alternating minimization algorithm (S-AMA)(Wang et al. 2018) to solve (B2).

Step3: For $k=1,\dots ,K$, update ${𝚿}_k^{\left(r\right)}$ by ${𝚿}_k^{\left(r\right)}={𝚿}_k^{(r-1)}+{𝚯}_k^{\left(r\right)}-{𝚯}_0-{𝚵}_k^{\left(r\right)}$.

Repeat the above steps until the termination criterion is met, which is defined as $\sum _{k=1}^K{‖{𝚯}_k^{\left(r\right)}-{𝚯}_k^{(r-1)}‖}_F/{‖{𝚯}_k^{(r-1)}‖}_F<{10}^{-2}$. This process yields the solution ${𝚯}^{\left(m\right)}=\left\{{𝚯}_k^{\left(m\right)},k=\right.1,\dots ,K\}$ for (B1). It is worth mentioning that the update in Step 1 inherently ensures the positive definiteness of the estimated precision matrices. However, the resulting solution may not always be symmetric. To address this, they can be symmetrized following the approach proposed by Cai et al. (2016):

$$\begin{gathered} {\theta }_{kij}^{\left(m\right)}={\theta }_{kij}^{\left(m\right)}I\left(\left|{\theta }_{kij}^{\left(m\right)}\right|⩽\left|{\theta }_{kji}^{\left(m\right)}\right|\right) \\ +{\theta }_{kji}^{\left(m\right)}I\left(\left|{\theta }_{kij}^{\left(m\right)}\right|>\left|{\theta }_{kji}^{\left(m\right)}\right|\right). \end{gathered}$$

Next, to implement the S-AMA, (B2) can be rewritten as

$$\begin{gathered} \underset{𝚵}{\text{argmin}}\frac{\kappa }{2}\sum _{j=1}^{p^2}{‖{\mathit{\xi }}_{\left(j\right)}-{\mathit{z}}_{\left(j\right)}‖}_2^2 \\ +\sum _{r\in ℰ}p\left({\left({\eta }_r^{\left(m\right)}+{‖{\mathit{v}}_r‖}_2^2\right)}^{1/2},{\lambda }_3\right) \\ +\sum _{j=1}^{p^2}\sum _{k=1}^{K}p\left(\left|{\xi }_{kj}\right|,{\lambda }_2\right)\cdot I\left(j\in 𝒪\right), \\ \mathit{\text{s.t.}}\text{vec}{𝚵}_k-\text{vec}{𝚵}_{k^{\prime }}-{\mathbf{v}}_r=0. \end{gathered}$$

where $ℰ=\left\{\left(k,k^{\prime }\right):1⩽k,k^{\prime }⩽K\right\},{\eta }_r^{\left(m\right)}=\Vert {\mathit{\mu }}_k^{\left(m\right)}-{\mathit{\mu }}_{k^{\prime }}^{\left(m\right)}{\Vert }_2^2,{\mathit{\xi }}_{\left(j\right)},{\mathit{z}}_{\left(j\right)}\in {\mathbb{R}}^K$ are the $j$-th columns of $(\text{vec}{𝚵}_1,\dots ,\text{vec}{\left.{𝚵}_K\right)}^{\top }$ and ${\left(\text{vec}{\mathcal{Z}}_1,\dots ,\text{vec}{\mathcal{Z}}_K\right)}^{\top }$, respectively, $j=1,\dots ,p^2,{\xi }_{kj}$ is the $k$ th element of ${\mathit{\xi }}_{\left(j\right)}$, and $𝒪=\{j:j\ne d(p+1)+1,d=0,1,\dots ,p-1\}$ is the index set of the off-diagonal components of the differential precision matrices. It is equivalent to minimizing the following augmented Lagrangian function:

$$\begin{gathered} {𝒬}_{{\kappa }^{\prime }}(𝚵,\mathit{V},\mathit{D})=\frac{\kappa }{2}\sum _{j=1}^{p^2}{‖{\mathit{\xi }}_{\left(j\right)}-{\mathit{z}}_{\left(j\right)}‖}_2^2 \\ +\sum _{r\in ℰ}p\left({\left({\eta }_r^{\left(m\right)}+{‖{\mathbf{v}}_r‖}_2^2\right)}^{1/2},{\lambda }_3\right) \\ +\sum _{j=1}^{p^2}\sum _{k=1}^{K}p\left(\left|{\xi }_{jk}\right|,{\lambda }_2\right)\cdot I(j\in 𝒪) \\ +\sum _{r\in ℰ}⟨{\mathit{d}}_r,{\mathbf{v}}_r-\text{vec}{𝚵}_k+\text{vec}{𝚵}_{k^{\prime }}⟩ \\ +\frac{{\kappa }^{\prime }}{2}\sum _{r\in ℰ}{‖{\mathbf{v}}_r-\text{vec}{𝚵}_k+\text{vec}{𝚵}_{k^{\prime }}‖}_2^2, \end{gathered}$$

where ${\kappa }^{\prime }$ is a small penalty parameter. In this study, we set ${\kappa }^{\prime }=1.\mathit{V}=\left({\mathbf{v}}_1,\dots ,{\mathbf{v}}_{\left|ℰ\right|}\right)$, and $\mathit{D}=\left({\mathit{d}}_1,\dots ,{\mathit{d}}_{\left|ℰ\right|}\right)$. S-AMA minimizes the augmented Lagrangian problem by alternatively solving one block of variables at a time. In the $l$–step of the S-AMA algorithm, we have

$$\begin{gathered} {𝚵}^{\left(l\right)}=\underset{𝚵}{\text{argmin}}{𝒬}_{{\kappa }^{\prime }}\left(𝚵,{\mathit{V}}^{(l-1)},{\mathit{D}}^{(l-1)}\right), \\ {\mathit{V}}^{\left(l\right)}=\underset{\mathit{V}}{\text{argmin}}{𝒬}_{{\kappa }^{\prime }}\left({𝚵}^{\left(l\right)},\mathit{V},{\mathit{D}}^{(l-1)}\right), \\ {\mathit{d}}_r^{\left(l\right)}={\mathit{d}}_r^{(l-1)}+{\kappa }^{\prime }\left({\mathbf{v}}_r^{\left(l\right)}-\text{vec}{𝚵}_k^{\left(l\right)}+\text{vec}{𝚵}_{k^{\prime }}^{\left(l\right)}\right) \\ \times I\left({‖{\mathbf{v}}_r^{\left(l\right)}‖}_2>0\right),r\in ℰ. \end{gathered}$$

It is noted that S-AMA differs from ADMM in the update of $𝚵$. Specifically, updating $𝚵$ requires solving $p^2$ individual penalization problems:

$$\underset{{\mathit{\xi }}_{\left(j\right)}}{\text{min}}\frac{\kappa }{2}{‖{\mathit{\xi }}_{\left(j\right)}-{\mathit{u}}_{\left(j\right)}‖}_2^2+\sum _{k=1}^{K}p\left(\left|{\xi }_{kj}\right|,{\lambda }_2\right)\cdot I(j\in 𝒪),$$ (B3)

where ${\mathit{u}}_{\left(j\right)}={\mathit{z}}_{\left(j\right)}+\sum _{r\in ℰ}{d}_{jr}\left({\mathit{e}}_k-{\mathit{e}}_{k^{\prime }}\right),d_{jr}$ is the $j$-th element of ${\mathit{d}}_r$, and ${\mathit{e}}_k$ is a $K$-dimensional vector with the $k$-th component being 1 and the other components being 0. For (B3), the MCP estimators have the following closed form:

$$\begin{gathered} {\widehat{\xi }}_{kj}=I(j\in 𝒪)\cdot \left\{\begin{array}{ll}\frac{ST\left(u_{kj},{\lambda }_2/\kappa \right)}{1-1/\left(a\kappa \right)}& \text{if}\left|u_{kj}\right|⩽a{\lambda }_2\\ u_{kj}& \text{if}\left|u_{kj}\right|>a{\lambda }_2\end{array}\right. \\ +I(j\notin 𝒪)\cdot u_{kj}, \end{gathered}$$

where $ST(c,\lambda )=\left(\right|c|-\lambda {)}_{+}\cdot \text{sign}(c)$. For $,{\mathbf{v}}_r$ ‘s can be updated separately. That is,

$${\widehat{\mathbf{v}}}_r=\underset{{\mathbf{v}}_r}{\text{argmin}}\frac{{\kappa }^{\prime }}{2}{‖{\mathbf{v}}_r-{\mathit{\omega }}_r‖}_2^2+p\left({\left({\eta }_r^{\left(m\right)}+{‖{\mathbf{v}}_r‖}_2^2\right)}^{1/2},{\lambda }_3\right),$$ (B4)

where ${\mathit{\omega }}_r=\text{vec}{𝚵}_k-\text{vec}{𝚵}_{k^{\prime }}-{\mathit{d}}_r/{\kappa }^{\prime }$. By the Karush-Kuhn-Tucker (KKT) conditions of the group MCP problem, the solution to (B4) has a closed form. If ${\left({\eta }_r^{\left(m\right)}+{‖{\mathit{\omega }}_r‖}_2^2\right)}^{1/2}⩽a{\lambda }_3$,

$$\begin{gathered} {\widehat{\mathbf{v}}}_r=\frac{{\mathit{\omega }}_r}{{\left({\eta }_r^{\left(m\right)}+{‖{\mathit{\omega }}_r‖}_2^2\right)}^{1/2}} \\ \times \frac{ST\left({\left({\eta }_r^{\left(m\right)}+{‖{\mathit{\omega }}_r‖}_2^2\right)}^{1/2},{\lambda }_3/{\kappa }^{\prime }\right)}{1-1/\left(a{\kappa }^{\prime }\right)}, \\ \text{If}{\left({\eta }_r^{\left(m\right)}+{‖{\mathit{\omega }}_r‖}_2^2\right)}^{1/2}>a{\lambda }_3, \\ {\widehat{\mathbf{v}}}_r={\left({\eta }_r^{\left(m\right)}+{‖{\mathit{\omega }}_r‖}_2^2\right)}^{1/2}. \end{gathered}$$

Data availability

The data that support the findings in this paper are openly available in https://toil-xena-hub.s3.us-east-1.amazonaws.com/download/TcgaTargetGtex_gene_expected_count.gz.

Code Availability

The R program implementing the proposed method is available at https://github.com/Linxi-Li/hetero-difgraph.git.