Hierarchical cancer heterogeneity analysis based on histopathological imaging features

Abstract

In cancer research, supervised heterogeneity analysis has important implications. Such analysis has been traditionally based on clinical/demographic/molecular variables. Recently, histopathological imaging features, which are generated as a byproduct of biopsy, have been shown as effective for modeling cancer outcomes, and a handful of supervised heterogeneity analysis has been conducted based on such features. There are two types of histopathological imaging features, which are extracted based on specific biological knowledge and using automated imaging processing software, respectively. Using both types of histopathological imaging features, our goal is to conduct the first supervised cancer heterogeneity analysis that satisfies a hierarchical structure. That is, the first type of imaging features defines a rough structure, and the second type defines a nested and more refined structure. A penalization approach is developed, which has been motivated by but differs significantly from penalized fusion and sparse group penalization. It has satisfactory statistical and numerical properties. In the analysis of lung adenocarcinoma data, it identifies a heterogeneity structure significantly different from the alternatives and has satisfactory prediction and stability performance.

1 |. INTRODUCTION

Cancer is intrinsically heterogeneous. Heterogeneity analysis has played a pivotal role in cancer research, and can be roughly classified as unsupervised and supervised. The two types of analysis have different implications and utilization (Wiwie et al., 2015). In this study, we conduct supervised analysis, which may be clinically more relevant by taking into account cancer outcomes/phenotypes. In “traditional” supervised cancer heterogeneity analysis, clinical, demographic, and molecular variables have been used. For examples, Xie et al. (2020) conduct supervised heterogeneity analysis of lung carcinoma with gene expression data. Liu and Sunil (2020) perform supervised therapeutic heterogeneity analysis of blood cancer with drug treatment, copy number variation, and genome-wide transcriptional data. An alternative source of data comes from histopathological imaging—a “byproduct” of biopsy, which is routinely ordered for cancer and suspicious patients. Compared to some other types of data, especially molecular data, histopathological imaging data are much more cost-effective and broadly available (Cooper et al., 2015). Histopathological images have been long used for definitive diagnosis and staging. Relatively recently, studies have shown that histopathological imaging features can also be effective for modeling cancer biomarkers, prognosis, and other outcomes (Yu et al., 2016; Luo et al., 2017; Choi and Na, 2018). Among them, a handful have conducted supervised and unsupervised heterogeneity analysis. For example, Luo et al. (2017) examine lung cancer prognosis and show that imaging features associated with overall survival differ significantly across subgroups. This study may be limited by the subgroup information derived from pathological category as opposed to data, and by the simple Cox regression. As another example, Choi and Na (2018) conduct conventional hierarchical clustering analysis of histopathological imaging features, and suggest that immune cell composition in tumor microenvironment is heterogeneous across lung adenocarcinomas and closely associated with tumor metabolism and patient prognosis. Such studies have established the feasibility and effectiveness of heterogeneity analysis for cancer risk, prognosis, and biomarkers based on histopathological imaging features.

Two types of histopathological imaging features have been used in cancer modeling. The first type has been inspired by visual attributes of images traditionally identified by pathologists as important for cancer grading and diagnosis. Such features are at the clinical level and require prior knowledge of human cancer (Gurcan et al., 2009). This type of imaging features has been shown as informative for modeling cancer outcomes, especially including those of lung cancer (Wang et al., 2018). In contrast, features of the second type are captured by computer algorithms in an unsupervised and automated manner. They contain pixel-, object-, and semantic-level information that can go beyond what pathologists can generate via manual examination. These higher resolution features are logical expansions and may improve on the first type of features (Kothari et al., 2013; Aeffner et al., 2019). They have also been analyzed in recent cancer modeling (Sun et al., 2018; He et al., 2020).

In this study, for extracting the first type of imaging features, we adopt the pipeline sketched in the upper panel of Figure 1 (this figure appears in color in the electronic version of this article, and any mention of color refers to that version). Briefly, this pipeline applies ConvPath (Wang et al., 2019), a deep learning cell classification tool, and first identifies tumor, stroma, and lymphocyte cells in images. Based on the cell type information, it extracts six features that describe the perimeter and area of each type of cell. Such features have direct biological interpretations. On the negative side, this pipeline demands first manually labeling images by pathologists and can be time- and labor-intensive. In addition, the number of extracted features is small, limiting the amount of available information. The pipeline for extracting the second type of imaging features is sketched in the lower panel of Figure 1, and involves slide chopping, random selection of subimages, feature extraction using digital imaging processing software (CellProfiler in this study, Carpenter et al. (2006)), and feature averaging. Imaging features generated by this pipeline rely less on input from pathologists and can have a higher dimensionality. They can potentially capture more and higher-resolution information, such as on local anatomical structure and more global patterns of tumor cell and tumor nuclei (Yu et al., 2016). On the negative side, the extracted features do not have direct biological interpretations and may contain noise.

FIGURE 1.

Pipelines for extracting the two types of histopathological imaging features

The goal of this study is to conduct supervised cancer heterogeneity analysis based on histopathological imaging features. Significantly different from the existing literature, we aim at utilizing both types of imaging features, which describe information contained in the same images using different techniques and from different angles. The scheme of our analysis is sketched in Figure 2A. With a group of subjects (upper panel), “ordinary” heterogeneity analysis using the first type of imaging features can separate them into subgroups (middle panel, where different colors represent different subgroups—this figure appears in color in the electronic version of this article, and any mention of color refers to that version). Here this type of imaging features is first considered as it has more direct biological interpretations. Then the question we seek to address is: can a finer sub-subgrouping structure as shown in the lower panel be further obtained? This is sensible as the second type of imaging features has a higher resolution and may contain finer information to identify more subtle sub-subgrouping structures. In the lower panel of Figure 2A, with the natural order of the two types of imaging features, the heterogeneity structure has a hierarchy, which, to the best of our knowledge, has not been studied in the literature.

FIGURE 2.

Analysis scheme and result from one simulated dataset: different colors represent different (sub-)subgroups. Type 1/2: using the two types of imaging features. (A) Upper: ungrouped subjects. Middle: with “ordinary” heterogeneity analysis and Type 1 imaging features, subjects are separated into two subgroups. Lower: with the proposed hierarchical heterogeneity analysis, subjects are separated into two subgroups based on Type 1 imaging features and then further separated into four sub-subgroups based on Type 2 imaging features. The lengths of the color bars are proportional to the sizes of (sub-)subgroups. (B) With one simulation replicate, subgrouping structures are identified by the proposed method and two alternatives (which generate the same subgrouping structures with the two types of imaging features). The same position on different bars represents the same sample

The idea of obtaining more refined cancer heterogeneity structures with the increased resolution of information/techniques is not new. Take breast cancer as an example. In the past, with information/technique limitations, it was considered as a single disease. With the development of high-throughput profiling and information contained in gene expressions, it was separated into five subtypes: Luminal A, Luminal B, HER2-enriched, Triple-negative, and Claudin-low (Prat et al., 2015). Further advancements in sequencing have suggested that these subtypes may contain finer structures. For example, a recent study (Gong et al., 2021) suggests that the Triple-Negative subtype can be further separated into three sub-subtypes (lipogenic, glycolytic, and mixed). In a sense, this study pursues the “histopathological imaging analogy” of such analysis.

For supervised heterogeneity analysis, there are multiple techniques, with finite mixture of regression (FMR) perhaps being the most popular (Khalili and Chen, 2007; Städler et al., 2010). A relatively more recent technique is penalized fusion (Ma and Huang, 2017; Chen et al., 2020). Compared to FMR and some other techniques, it determines the number (and hence structure) of subgroups in a less ad hoc way. It can in principle accommodate subgroups as small as size one. Its competitive statistical and numerical properties have been well established (Wang et al., 2018; Yang et al., 2019). It is especially noted that penalized fusion has been applied to cancer studies and histopathological imaging data (He et al., 2020). To more intuitively demonstrate the different working characteristics of the proposed and alternative methods, we simulate one data set with the true heterogeneity structure shown in the lower panel of Figure 2A. In Figure 2B, we show the estimated heterogeneity structures using the proposed method, a direct application of penalized fusion, and FMR. The differences are obvious, and the proposed approach can more satisfactorily identify the hierarchical heterogeneity structure.

This study can complement and advance from the existing ones in the following important aspects. First, it advances from the existing heterogeneity analysis by introducing the novel and sensible hierarchical structure, which can provide information on the similarity of subjects at multiple levels. It is noted that this hierarchical heterogeneity structure and hence proposed analysis are also applicable to other settings, as long as an order of variables can be defined. Second, a novel analysis approach is developed. It is built on but significantly advances from penalized fusion and sparse group penalization and can generate the sophisticated heterogeneity structure as sketched in the lower panel of Figure 2A in a single estimation. Third, this study provides an alternative way of conducting cancer heterogeneity analysis, which includes some existing analyses as special cases. Last but not least, it may also provide additional insights into the heterogeneous associations between histopathological imaging features and an important lung cancer biomarker.

2 |. METHODS

Consider n independent subjects with measurements ${\left\{y_i,x_i,z_i\right\}}_{i=1}^n$, where x_i = (x_i1, … , x_iq)^⊤ and z_i = (z_i1, … , z_ip)^⊤. Here for subject i, y_i is the response variable, and x_i and z_i are the first and second type of imaging features, respectively. Consider the heterogeneity model:

$$y_i=x_i^{\top }{\beta }_i+z_i^{\top }{\gamma }_i+{ϵ}_i,i=1,\dots ,n,$$

where β_i and γ_i are the q- and p-dimensional vectors of unknown regression coefficients, respectively, and ϵ_i is the random error with E(ϵ_i) = 0 and Var(ϵ_i) = σ². Intercept is omitted for the simplicity of notation. We consider linear regression for a continuous response, which matches the data analysis in Section 4. Note that the proposed approach is potentially applicable to other types of response/model. Each subject is flexibly modeled to have its own regression coefficients, and two subjects belong to the same (sub)subgroup if and only if they have the same regression model/coefficients.

Significantly advancing from the existing literature, we consider a more sophisticated heterogeneity structure as sketched in the lower panel of Figure 2A, where β_i’s define a “rough” heterogeneity structure with K₁ subgroups, and γ_i’s define a more refined heterogeneity structure with K₂ sub-subgroups. Denote $\left\{{\mathcal{G}}_1^{*},\dots ,{\mathcal{G}}_{K_1}^{*}\right\}$ as the collection of subject index sets of the K₁ subgroups, and $\left\{{\mathcal{T}}_1^{*},\dots ,{\mathcal{T}}_{K_2}^{*}\right\}$ as the collection of subject index sets of the K₂ sub-subgroups. The hierarchy of heterogeneity amounts to a nested structure. That is, there exists a mutually exclusive partition of $\left\{1,\dots ,K_2\right\}:\left\{{\mathcal{H}}_1,\dots ,{\mathcal{H}}_{K_1}\right\}$ satisfying ${\mathcal{G}}_{k_1}^{*}={\cup }_{k_2\in {\mathcal{H}}_{k_1}}{\mathcal{T}}_{k_2}^{*}$, 1 ⩽ k₁ ⩽ K₁, 1 ⩽ k₂ ⩽ K_2.

2.1 |. Penalized estimation

For simultaneous estimation and determination of the heterogeneity structure, we propose the penalized objective function:

$$\begin{gathered} Q(\beta ,\gamma )=\frac{1}{2}\sum _{i=1}^n{\left(y_i-x_i^{\top }{\beta }_i-z_i^{\top }{\gamma }_i\right)}^2 \\ +\sum _{1⩽j<m⩽n}p\left(\sqrt{{\Vert {\beta }_j-{\beta }_m\Vert }_2^2+{\Vert {\gamma }_j-{\gamma }_m\Vert }_2^2},{\lambda }_1\right) \\ +\sum _{1⩽j<m⩽n}p\left({\Vert {\beta }_j-{\beta }_m\Vert }_2,{\lambda }_2\right), \end{gathered}$$ (2.1)

where $\beta ={\left({\beta }_1^{\top },\dots ,{\beta }_n^{\top }\right)}^{\top }$, $\gamma ={\left({\gamma }_1^{\top },\dots ,{\gamma }_n^{\top }\right)}^{\top }$, and p(·, λ) is a concave penalty function with tuning parameter λ > 0. In our numerical study, we adopt MCP (Minimax Concave Penalty; Zhang (2010)), and note that SCAD (Smoothly Clipped Absolute Deviation Penalty) and some other penalties are also applicable. Consider $(\widehat{\beta },\widehat{\gamma })=\underset{\beta ,\gamma }{\text{argmin}}Q(\beta ,\gamma )$. Denote $\left\{{\widehat{\alpha }}_1,\dots ,{\widehat{\alpha }}_{{\widehat{K}}_1}\right\}$ and $\left\{{\widehat{\delta }}_1,\dots ,{\widehat{\delta }}_{{\widehat{K}}_2}\right\}$ as the distinct values of $\widehat{\beta }$ and $\widehat{\gamma }$, respectively. Then $\left\{\widehat{{\mathcal{G}}_1},\dots ,{\widehat{\mathcal{G}}}_{{\widehat{K}}_1}\right\}$ and $\left\{{\widehat{\mathcal{T}}}_1,\dots ,{\widehat{\mathcal{T}}}_{{\widehat{K}}_2}\right\}$ constitute mutually exclusive partitions of {1, … , n}, where ${\widehat{\mathcal{G}}}_{k_1}=\left\{i:{\widehat{\beta }}_i={\widehat{\alpha }}_{k_1},i=1,\dots ,n\right\}$, $k_1=1,\dots ,{\widehat{K}}_1$ and ${\widehat{\mathcal{T}}}_{k_2}=\left\{i:{\widehat{\gamma }}_i={\widehat{\delta }}_{k_2},i=1,\dots ,n\right\}$, $k_2=1,\dots ,{\widehat{K}}_2$. Collectively, they fully determine the heterogeneity structure.

Rationale In the objective function, the first term measures lack of fit (and can be replaced with the negative log likelihood for another type of data/model). The proposed penalty is built on the penalized fusion technique. For the proposed regression models, under the standard penalized fusion (Ma and Huang, 2017), the penalty is ${\sum }_{1⩽j<m⩽n}p\left(\parallel {\left({\beta }_j^{\top },{\gamma }_j^{\top }\right)}^{\top }-{\left({\beta }_m^{\top },{\gamma }_m^{\top }\right)}^{\top }{\parallel }_2\right)$. It can shrink some differences to exactly zero, and subjects with exactly the same regression coefficients are assigned to the same subgroup. However, using the standard penalized fusion, different types of variables can only produce the same subgrouping structure. Beyond penalized fusion, the proposed penalty also shares some similar spirit with sparse group penalization. That is, a hierarchical penalty is imposed on the two types of variables, achieving a nested subgrouping structure. The resulted estimates have the following possibilities: (a) ${\widehat{\beta }}_j={\widehat{\beta }}_m$ and ${\widehat{\gamma }}_j={\widehat{\gamma }}_m$. Then subjects j and m are assigned to the same subgroup (of the rough heterogeneity structure) and the same sub-subgroup (of the finer heterogeneity structure); (b) ${\widehat{\beta }}_j\ne {\widehat{\beta }}_m$ and ${\widehat{\gamma }}_j\ne {\widehat{\gamma }}_m$. Then subjects j and m are assigned to different subgroups and different sub-subgroups; and (c) ${\widehat{\beta }}_j\ne {\widehat{\beta }}_m$ but ${\widehat{\gamma }}_j\ne {\widehat{\gamma }}_m$. Then subjects j and m are assigned to the same subgroup but different sub-subgroups. Note that the term related to γ_j − γ_m only appears in the first group penalty. Due to the “all in or all out” property of group penalization, the case with ${\widehat{\beta }}_j\ne {\widehat{\beta }}_m$ and ${\widehat{\gamma }}_j\ne {\widehat{\gamma }}_m$ cannot happen. This approach has a mechanism similar to the hierarchical structured sparsity penalty (Bien et al., 2013).

Combining the above scenarios, we see that, if subjects j and m are assigned to the same sub-subgroup, then they must be assigned to the same subgroup. On the other hand, when subjects j and m are assigned to the same subgroup, they can be assigned to the same sub-subgroup or different sub-subgroups. As such, the proposed approach can achieve the nested structure sketched in Figure 2A.

We note that, in some studies, the second type of imaging features is high dimensional and noisy. Directly analyzing such data demands imposing additional sparsity or low-rank (Jing et al., 2018) constraints. We have very briefly explored such possibilities and found that they make computation intractable and also introduce tremendous theoretical challenges. In this study, we focus on the scenario where the second type of imaging features is low dimensional and noise is not of a serious concern. If needed, this can be achieved with prescreening, which is a popular technique and adopted in our data analysis.

2.2 |. Statistical properties

Denote the true values of parameters as ${\beta }^{*}={\left({\beta }_1^{*\top },\dots ,{\beta }_n^{*\top }\right)}^{\top }$ and ${\gamma }^{*}={\left({\gamma }_1^{*\top },\dots ,{\gamma }_n^{*\top }\right)}^{\top }$. Define the minimal differences of the common values between two (sub-)subgroups as

$$\begin{gathered} b_1=\underset{j\in {\mathcal{T}}_k^{*},m\in {\mathcal{T}}_{k^{\prime }}^{*},1⩽k\ne k^{\prime }⩽K_2}{\text{min}}{\Vert {\gamma }_j^{*}-{\gamma }_m^{*}\Vert }_2,b_2 \\ =\underset{j\in {\mathcal{G}}_k^{*},m\in {\mathcal{G}}_{k^{\prime }}^{*},1⩽k\ne k^{\prime }⩽K_1}{\text{min}}{\Vert {\beta }_j^{*}-{\beta }_m^{*}\Vert }_2. \end{gathered}$$

Denote $\left|{\mathcal{T}}_{\text{min}}\right|=\underset{1⩽k_2⩽K_2}{\text{min}}\left|{\mathcal{T}}_{k_2}^{*}\right|$, where |·| is the cardinality of the set. Define the 2-norm of the q-dimensional vector d = (d₁, … , d_q)^⊤ as $\parallel d{\parallel }_2=\sqrt{{\sum }_{j=1}^q{d}_j^2}$. Let ψ_min(A) denote the smallest eigenvalues of matrix A. Mild assumptions are made and described in the Supporting Information, with which we can establish the following consistency properties.

Theorem 1. Suppose that Conditions A.1–A.3 (Supplementary Information) hold. Assume that $\left|{\mathcal{T}}_{\text{min}}\right|\gg \sqrt{\left(K_{1}q+K_{2}p\right)n\text{\hspace{0.17em}log\hspace{0.17em}}n}$, b₁ > (a + 0.5)λ₁, b₂ > (a + 0.5)λ₂, where a is the regularization parameter of the concave penalty defined in Condition A.1, and $\text{min}\left\{{\lambda }_1,{\lambda }_2\right\}\gg {\left|{\mathcal{T}}_{\text{min}}\right|}^{-1}\sqrt{\left(K_{1}q+K_{2}p\right)n\text{\hspace{0.17em}log\hspace{0.17em}}n}$. Then we have:

1. (Consistency of identifying the number of (sub-)subgroups) $pr\left({\widehat{K}}_1=K_1,{\widehat{K}}_2=K_2\right)\to 1$.

2. (Consistency of subgrouping) $\text{pr}({\widehat{\mathcal{G}}}_{k_1}={\mathcal{G}}_{k_1}^{*},\text{ for }{k}_1=1,\dots ,K_1)\to 1$ and $\text{pr}\left({\widehat{\mathcal{T}}}_{k_2}={\mathcal{T}}_{k_2}^{*}\text{, for }{k}_2=1,\dots ,K_2\right)\to 1$.

3. (Rate of convergence) There exists a local minimum of (2.1) that satisfies:

   $$\begin{gathered} \underset{i}{\text{sup}}{\Vert {\widehat{\beta }}_i-{\beta }_i^{*}\Vert }_2+\underset{i}{\text{sup}}{\Vert {\widehat{\gamma }}_i-{\gamma }_i^{*}\Vert }_2 \\ =O_p\left({\left|{\mathcal{T}}_{\text{min}}\right|}^{-1}\sqrt{\left(K_{1}q+K_{2}p\right)n\text{\hspace{0.17em}log\hspace{0.17em}}n}\right). \end{gathered}$$

This theorem shows that the proposed approach has much-desired consistency properties. In particular, it can accurately estimate the number of (sub-)subgroups and recover the subgrouping structures with a high probability. Such results, particularly the consistency of the number of (sub-)subgroups, can be extremely difficult to establish with some other heterogeneity analysis techniques. This approach also has the “ordinary” estimation consistency. With the special design of the penalty, it easily follows that the heterogeneity hierarchy holds. Proof is provided in the Supporting Information. We note that although certain steps have roots in Ma and Huang (2017), Yang et al. (2019) and other published studies, as the penalty is significantly different, the theoretical development is nontrivial. For example, more complex consistency conditions on subgrouping recovery theory need to be established under the hierarchical nested penalty.

2.3. Computation

We derive an ADMM (Alternating Direction Method of Multipliers) algorithm for optimizing (2.1). Some developments are specific to MCP, and optimizing with other penalties demands minor modifications. The objective function can be reformulated as

$$\begin{gathered} {\mathcal{L}}_0(\beta ,\gamma ,v,\omega )=\frac{1}{2}\sum _{i=1}^n{\left(y_i-x_i^{\top }{\beta }_i-z_i^{\top }{\gamma }_i\right)}^2 \\ +\sum _{1⩽j<m⩽n}p\left(\sqrt{{\Vert {\mathit{v}}_{jm}\Vert }_2^2+{\Vert {\mathit{\omega }}_{jm}\Vert }_2^2},{\lambda }_1\right) \\ +\sum _{1⩽j<m⩽n}p\left({\Vert {\mathit{v}}_{jm}\Vert }_2,{\lambda }_2\right), \end{gathered}$$

subject to υ_jm = β_j − β_m, ω_jm = γ_j − γ_m, where $v={\left\{v_{jm}^{\top },j<m\right\}}^{\top }$ and $\omega ={\left\{{\omega }_{jm}^{\top },j<m\right\}}^{\top }$. By the augmented Lagrangian method, the estimates can be obtained by minimizing:

$$\begin{gathered} \mathcal{L}(\beta ,\gamma ,v,\omega ,\xi ,\eta )={\mathcal{L}}_0(\beta ,\gamma ,v,\omega ) \\ +\sum _{j<m}{\xi }_{jm}^{\top }\left({\beta }_j-{\beta }_m-v_{jm}\right) \\ +\frac{\kappa }{2}\sum _{j<m}{\Vert {\beta }_j-{\beta }_m-v_{jm}\Vert }_2^2 \\ +\sum _{j<m}{\mathit{\eta }}_{jm}^{\top }\left({\gamma }_j-{\gamma }_m-{\omega }_{jm}\right) \\ +\frac{\kappa }{2}\sum _{j<m}{\Vert {\gamma }_j-{\gamma }_m-{\omega }_{jm}\Vert }_2^2, \end{gathered}$$

where the dual variables $\xi ={\left\{{\xi }_{jm}^{\top },j<m\right\}}^{\top }$ and $\eta ={\left\{{\eta }_{jm}^{\top },j<m\right\}}^{\top }$ are the Lagrange multipliers, ξ_jm and η_jm are q- and p-dimensional vectors, and κ is a penalty parameter.

Overall, we propose an iterative algorithm, with the FMR estimates as the initial for ${\left({\beta }^{(0)\top },{\gamma }^{(0)\top }\right)}^{\top }$, $v_{jm}^{(0)}={\beta }_j^{(0)}-{\beta }_m^{(0)},{\omega }_{jm}^{(0)}={\gamma }_j^{(0)}-{\gamma }_m^{(0)}$, ξ⁽⁰⁾ = 0, and η⁽⁰⁾ = 0. As iteration t + 1, the updates are

$$\left({\beta }^{(t+1)},{\gamma }^{(t+1)}\right)=\underset{\beta ,\gamma }{\text{argmin}}\mathcal{L}\left(\beta ,\gamma ,v^{(t)},{\omega }^{(t)},{\xi }^{(t)},{\eta }^{(t)}\right),$$ (2.2)

$$\left(v^{(t+1)},{\omega }^{(t+1)}\right)=\underset{v,\omega }{\text{argmin}}\mathcal{L}\left({\beta }^{(t+1)},{\gamma }^{(t+1)},v,\omega ,{\xi }^{(t)},{\eta }^{(t)}\right),$$ (2.3)

$${\xi }_{jm}^{(t+1)}={\xi }_{jm}^{(t)}+\kappa \left({\beta }_j^{(t+1)}-{\beta }_m^{(t+1)}-v_{jm}^{(t+1)}\right),$$ (2.4)

$${\eta }_{jm}^{(t+1)}={\eta }_{jm}^{(t)}+\kappa \left({\gamma }_j^{(t+1)}-{\gamma }_m^{(t+1)}-{\omega }_{jm}^{(t+1)}\right).$$ (2.5)

Details of (2.2) and (2.3) are provided in the Supporting Information. These updates are repeated until convergence, which is concluded when the difference between two consecutive estimates is smaller than a predefined cut-off. Convergence of this algorithm follows from Ma and Huang (2017) and is achieved in all of our numerical analyses. This algorithm adopts ADMM, which is similar to Ma and Huang (2017). However, it significantly differs in solving (2.3) by involving the complex hierarchical groupwise thresholding operator. Denote C_n = log[n(p + q)]. To select (λ₁, λ₂), following Ma and Huang (2017), we conduct a grid search and minimize the modified BIC-type criterion:

$$\begin{gathered} \text{BIC}\left({\lambda }_1,{\lambda }_2\right)=\text{log}\left[\frac{1}{n}\sum _{i=1}^n{\left\{y_i-x_i^{\top }{\widehat{\beta }}_i\left({\lambda }_1,{\lambda }_2\right)-z_i^{\top }{\widehat{\gamma }}_i\left({\lambda }_1,{\lambda }_2\right)\right\}}^2\right] \\ +C_n\frac{\text{log\hspace{0.17em}}n}{n}\left\{{\widehat{K}}_1\left({\lambda }_1,{\lambda }_2\right)q+{\widehat{K}}_2\left({\lambda }_1,{\lambda }_2\right)p\right\}. \end{gathered}$$

Here it is noted that, although this criterion has been shown as numerically satisfactory, its theoretical properties, for example, uniqueness of optimizer, remains unestablished.

3. SIMULATION

For n = 120 independent subjects, we generate x_i’s from a q-dimensional normal distribution N(0, Σ₁), where Σ₁ = (σ_1jm)_1⩽j,m⩽p with σ_1jm = 1 if j = m, and =0.2 otherwise. Then we randomly select 0.4q components and dichotomize at zero, leading to 0.6q continuous and 0.4q binary components. We simulate z_i’s from a p-dimensional normal distribution N(0, Σ₂). Two structures of the covariance matrix Σ = (σ_2jm)_1⩽j,m⩽p are considered. The first structure is auto-regressive (AR) correlation given by σ_2jm = ρ^|j−m| with ρ = 0.25 and 0.75 (denoted as AR1 and AR2, respectively). The second structure is banded correlation. Here two scenarios are considered and denoted as B1 and B2 respectively. Under B1, σ_2jm = I(j = m) + 0.3I(|j − m| = 1). Under B2, σ_2jm = I(j = m) + 0.6I(|j − m| = 1) + 0.3I((|j − m| = 2) Then We randomly select 0.4p components and dichotomize at zero. For the correlation between x_i and z_i, we consider the uncorrelated and weakly correlated structures, under which the correlation coefficients between the jth component of x_i and the mth component of z_i are 0 and 0.05min{min_1⩽j,m⩽p(σ_1jm), min_1⩽j,m⩽p(σ_2jm)}, respectively. We note that the imaging features analyzed in the next section are all continuous. Here we generate covariates that also have binary components to be more general, as the proposed approach is also applicable to other data scenarios that may have discrete components. We have experimented with all covariates being continuous and observed similar patterns as described below.

There are K₁ = 2 subgroups with β_i equal to (μ, μ, 0_q−2) and (−μ, −μ, 0_q−2), respectively. There are K₂ = 4 sub-subgroups with γ_i equal to (1.5μ, 1.5μ, 1.5μ, 1.5μ, 0_p−4), (0.5μ, 0.5μ, 0.5μ, 0.5μ, 0_p−4), (−0.5μ, −0.5μ, −0.5μ, −0.5μ, 0_p−4), and (−1.5μ, −1.5μ, −1.5μ, −1.5μ, 0_p−4), respectively. We set q = 5 and p = 10 and consider two signal levels with μ = 2 and 0.6. For the proportions of subjects in the sub-subgroups, we consider (1/4,1/4,1/4,1/4) and (1/6,1/6,1/3,1/3), which represent balanced and imbalanced designs. The random errors are generated from N(0, 0.5). For each simulation setting, we generate 100 replicates.

To gauge performance of the proposed approach, we consider the following competitors: (a) the “ordinary” penalized fusion approach (denoted as fusion1) as developed in Ma and Huang (2017); (b) the one-step FMR approach (denoted as FMR1), which applies the FMR technique to y_i ~ x_i + z_i, i = 1, … , n; (c) the one-step response-based clustering approach (respclust1), which first clusters the subjects based on the response variable, and then applies “linear regression + MCP” to each subgroup. Here we note that penalization is not necessary. It is applied to be more coherent with the proposed penalized estimation; (d) the one-step residual-based clustering (resiclust1), which applies “linear regression + MCP” under the homogeneity assumption $y_i=x_i^{\top }\beta +z_i^{\top }\gamma +{ϵ}_i$, groups subjects based on the residuals $y_i-x_i^{\top }\widehat{\beta }+z_i^{\top }\widehat{\gamma }$, and then applies “linear regression + MCP” again to each subgroup. The above approaches can only generate one-level heterogeneity structures; (e) the two-step FMR approach (FMR2), which first applies FMR to y_i ~ x_i + z_i, i = 1, … , n, and then applies FMR again to $y_i-x_i^{\top }{\widehat{\beta }}_i~z_i$ within each subgroup. By repeatedly applying the FMR technique, this approach can generate two-level heterogeneity structures, which can be more comparable to the proposed approach; (f) the two-step response-based clustering (respclust2), which first clusters all subjects based on the response variable, and then clusters the subjects within each subgroup using the same technique; and (g) the two-step residual-based clustering (resiclust2), which is similar to (f) but is based on residuals as in (d). For fusion1, the number of subgroups is determined in a similar way as for the proposed approach. For approaches (b), (c), and (d), the number of subgroups is selected from {2, 3, 4, 5, 6}. For approaches (e), (f), and (g), the number of (sub)-subgroups is selected from {2, 3}. It is noted that, comparatively, the proposed approach can much more easily identify the number of (sub)-subgroups. To the best of our knowledge, there is no existing approach that directly generates heterogeneity structures similar to the proposed one.

The following measures are adopted to evaluate performance: (a) percentage of ${\widehat{K}}_{1,2}$ equal to K_1,2 (denoted by “per”), (b) mean and sd of ${\widehat{K}}_{1,2}$, (c) subgrouping consistency defined by

$$\begin{gathered} \text{SC}(\widehat{\phi },\phi ):={\left(\begin{array}{l}n\hfill \\ 2\hfill \end{array}\right)}^{-1}\mid \{(i,j):I\left(\widehat{\phi }\left(x_i,z_i\right)=\widehat{\phi }\left(x_j,z_j\right)\right) \\ =I\left(\phi \left(x_i,z_i\right)=\phi \left(x_j,z_j\right)\right);i<j\}\mid , \end{gathered}$$

where $\widehat{\phi }$ and φ are the estimated and true (sub)subgrouping memberships, respectively. This is essentially the Rand index (Rand, 1971), which has been commonly adopted to measure clustering accuracy. In particular, it measures the percentage of correct decisions (if pairs of subjects in different subgroups are assigned to different subgroups, and pairs from the same subgroup are assigned to the same subgroup) out of a total of $\left(\begin{array}{l}n\hfill \\ 2\hfill \end{array}\right)$ possible decisions. It is calculated for the two heterogeneity levels separately. We also consider (d) mean squared error (MSE) for β and γ: $\text{MSE}(\beta )=\sqrt{\frac{{{\sum }_{i=1}^n\Vert {\widehat{\mathit{\beta }}}_i-{\beta }_i\Vert }_2^2}{nq}}$ and $\text{MSE}(\gamma )=\sqrt{\frac{{{\sum }_{i=1}^n\Vert {\widehat{\gamma }}_i-{\gamma }_i\Vert }_2^2}{np}}$.

Penalized fusion is usually computationally expensive. However, simulation suggests that that proposed analysis is still feasible. For analyzing one replicate, it takes about 15 minutes on a laptop with regular configurations. To gain more insight into the operating characteristics of the proposed approach, in the Supporting Information, we show the modified BIC criterion (Figure C1), estimated K₁ and K₂ (Figure C2), and estimates for one component of β and one component of γ (Figure C3), as a function of λ₁ and λ₂, for one simulated data set. In Figure C1, there is one unique optimizer. The paths are “well-behaved” and similar to those of other penalized estimations.

Summary results for the scenario with correlated x_i and z_i, balanced design, and μ = 2 are provided in Figure 3. The rest of the results are provided in Figures C4– C10 (Supporting Information). The corresponding numerical values are provided in Tables C1– C8 (Supporting Information). Across the whole spectrum of simulation, the proposed approach is observed to have highly competitive performance. Consider, for example, Figure 3 and the scenario with correlated x_i and z_i, balanced design, μ = 2, and AR1 correlation. For the heterogeneity defined associated with x_i and z_i, respectively, the proposed approach has per values 0.88 and 0.80, suggesting high accuracy. In contrast, all the alternatives fail in this aspect, in particular with respect to z_i. Overall, the proposed approach also has high SC values, although it is noted that for some scenarios, it not necessarily has the highest values (but very close). It is also observed to have high estimation accuracy. For example, for estimating β, the MSE values are 0.970 (fusion1), 1.430 (FMR1), 1.617 (repclust1), 1.860 (resiclust1), 1.423 (FMR2), 1.157 (respclust2), 1.310 (resiclust2), and 0.942 (proposed).

FIGURE 3.

Simulation results for the scenario with correlated x_i and z_i, balanced design, and μ = 2. Correlation structures from top to bottom: AR1, AR2, B1, and B2. This figure appears in color in the electronic version of this article, and any mention of color refers to that version

3.1 |. Additional exploration

Although the heterogeneity hierarchy is biologically and statistically sensible, it may still be of interest to explore what may happen when it is violated. In the middle panel of Figure C11 (Supporting Information), we sketch the setting where the heterogeneity structures defined by β and γ violate the hierarchy. Examining the proposed penalty and theoretical development suggests that our approach will generate a heterogeneity structure as represented in the lower panel of Figure C11. However, the estimation consistency may still hold. To explore this numerically, we simulate data with K₁ = 2 subgroups based on x and K₂ = 3 sub-subgroups based on z. Overall, four subject groups can be obtained with the combination of these two. The regression coefficients are (2, 2, 0_q−2, 3, 3, 3, 3, 0_p−4), (2, 2, 0_p+q−2), (−2, −2, 0_p+q−2), and (−2, −2, 0_q−2, −3, −3, −3, −3, 0_p−4). Other settings are the same as those described above. The results are summarized in Figure C12 and Table C9 (Supporting Information). As expected, the subgrouping accuracy is unsatisfactory. However, the proposed approach still has competitive estimation performance. In practical data analysis, the similarity of regression coefficients of different sub-subgroups may hint a violation of the hierarchy assumption.

4 |. DATA ANALYSIS

The Cancer Genome Atlas (TCGA) is a collective effort organized by the NIH and has published high-quality clinical, molecular, and imaging data on multiple cancer types. Compared to clinical and molecular data, the TCGA histopathological imaging data has been less analyzed. Studies based on the TCGA imaging data include Yu et al. (2017), Sun et al. (2018), Wang et al. (2018), and others, which adopt somewhat simple analysis methods. Here we analyze data on lung adenocarcinoma (LUAD), which are publicly available at the TCGA data portal (TCGA, 2021). The heterogeneity of lung cancer and possibility of studying such heterogeneity based on imaging features have been established in Luo et al. (2017) and other published studies. The response variable is FEV1, which is the percentage comparison to a normal value reference range of the volume of air that a patient can forcibly exhale from the lungs in one second pre-bronchodilator. It is an important indicator of lung function and an established biomarker for lung cancer prognosis and other outcomes. It is correlated with its post-bronchodilator counterpart, which is also an important biomarker for lung cancer capacity and prognosis. It is noted that the post-bronchodilator measure is often not measured in population studies (Mannino et al., 2011; Balte et al., 2020). The histogram in Figure C13 (Supporting Information) suggests a mixture distribution (heterogeneity).

In the TCGA study, whole-slide histopathological images and tissue images are captured at 20* or 40* magnification by the Aperio medical scanner and saved in the svs format. The pipeline for extracting the first type of imaging features has been briefly described in Section 1 and Figure 1. We refer to Wang et al. (2019) for more detailed information. Information on the six extracted features is available in Table 1. In the extraction of the second type of imaging features, in slide chopping, the subimages have 500×500 pixels, and 20 subimages are randomly selected for downstream analysis. Note that this is the common practice in published studies (Yu et al., 2017) and balances computational efficiency and effectiveness, and that these parameters are consistent with the literature (Zhu et al., 2016; Luo et al., 2017). There are 229 features available for analysis, most of which are not expected to be informative (Yu et al., 2017; Zhong et al., 2019). As such, we conduct a supervised prescreening, with FEV1 as the response variable, using the ridge fusion-based regression. This technique modifies the pairwise fusion approach by revising the penalty to ridge (Ma and Huang, 2017) and can be computationally more efficient. Then the top 10 features (Table 1), ranked based on residuals, are selected for downstream analysis, which can significantly reduce computational cost. Sensitivity analysis on the number of Type 2 features is performed (Table C10, Supporting Information), and it is observed that the (sub-)subgrouping structure is not sensitive to this choice. Overall, the analyzed data contains the six Type 1 imaging features, 10 Type 2 imaging features, and response variable for 118 subjects.

Table 1.

Data analysis using the proposed approach: estimated coefficients

Subgroup
Type 1 imaging featurea	1	2	3
LymphocytesPN	−3.9241	−1.3239	0.9290
StromaPN	0.6681	−2.1004	0.4450
TumorPN	−0.3196	1.4985	−0.3917
LymphocytesSN	2.3989	1.9573	−0.9580
StromaSN	−0.5659	0.9379	0.0290
TumorSN	−0.8804	−0.6502	−0.2264

Sub-subgroup
Selected Type 2 imaging feature	1–1	2–1	3–1	3–2
AreaShape-Center-Y	−0.2939	0.8353	0.0059	0.0052
AreaShape-Zernike-8–2	−0.8493	1.1858	−0.3053	0.1607
Granularity-12-ImageAfterMath	−0.8566	0.6822	−0.0904	−0.0731
Texture-Contrast-maskosingray-3–03	−1.1070	−0.1602	0.3224	1.7454
Texture-Correlation-maskosingray-3–01	−0.7805	−0.1187	0.0598	0.9032
Texture-DifferenceVariance-ImageAfterMath-3–03	−1.0674	0.0939	−0.1574	2.0068
Texture-SumVariance-ImageAfterMath-3–00	−0.1428	0.5558	−0.3443	0.4095
Texture-SumVariance-maskosingray-3–01	1.0721	−0.1465	−0.1281	−3.7819
Threshold-FinalThreshold-Identifyhemasub2	−0.0651	−1.0249	−0.3460	−2.3063
Threshold-SumOfEntropies-Identifyeosinprimarycytoplasm	−0.2158	−0.4746	−0.0795	−2.5133

^aLymphocytesPN: Perimeter of lymphocyte cell region/square root of image size. StromaPN: Perimeter of stromal cell region/square root of image size. TumorPN: Perimeter of tumor cell region/square root of image size. LymphocytesSN: Size of lymphocyte cell region/image size. StromaSN: Size of stromal cell region/image size. TumorSN: Size of tumor cell region/image size.

The analysis results are summarized in Table 1. Three subgroups are identified, with sizes 23, 23, and 72, respectively. More detailed information on which subjects belong to these subgroups is available from the authors. The three subgroups have significantly different estimated regression coefficients. With the second type of imaging features, the third subgroup is further split into two sub-subgroups, with sizes 49 and 23, respectively. This shows that the proposed approach does not force further splitting, as for the first two subgroups. The coefficients for sub-subgroups 3–1 and 3–2 are significantly different, suggesting no obvious violation of the hierarchy.

It has been recognized in the literature that the biological implications of histopathological imaging features, especially Type 2, are not clear. As such, we are not able to further interpret the heterogeneous regression models. To gain “indirect support,” we compare across the identified (sub-)subgroups samples’ clinical features, including pre- and post-bronchodilator FEV1, carbon monoxide diffusion, tumor size, disease free survival, and overall survival in Figure C14, as well as smoking status and tumor site in Table C11, in Supporting Information. Significant differences are observed, suggesting that the identified (sub-)subgroups may have clinically meaningful differences. It is observed that most of the relative differences in clinical outcomes are larger than 50% or even 100%, which indicates that they have significant differences across (sub-)subgroups. Moreover, the three subgroups differ significantly in smoking history. The difference in tumor site is significant between sub-subgroups 3–1 and 3–2. It is noted that, except for pre-bronchodilator FEV1, these clinical features are not included in model fitting. As such, there is no concern on over-fitting. This analysis can support the validity of the identified heterogeneity structure to a great extent.

4.1 |. Analysis with alternative methods

Data are also analyzed using the alternative methods. The estimation results are provided in Tables C12– C18 (Supporting Information). It is observed that different approaches lead to significantly different heterogeneity structures and estimates. In Table C19, the concordance values between the different heterogeneity structures are shown. The proposed approach has a higher concordance with fusion1, which is as expected since they have similar analysis schemes. It has moderate concordance with the other approaches.

In evaluation, we first remove one subject and then conduct estimation. Prediction is made for the one removed subject, we use the model that most of the removed subject’s co-members (in the analysis using the whole data) have for prediction. More specifically, denote the subject index sets of the K⁽⁰⁾ sub-subgroups in the analysis using the whole data as $\left\{{\mathcal{T}}_1^{(0)},\dots ,{\mathcal{T}}_{K^{(0)}}^{(0)}\right\}$, and the subject index sets of the K^(l) sub-subgroups in the analysis after removing the lth subject as $\left\{{\mathcal{T}}_1^{(l)},\dots ,{\mathcal{T}}_{K^{(l)}}^{(l)}\right\}$. For any l, there exists K⁽⁰⁾ satisfying $l\in {\mathcal{T}}_{k^{(0)}}^{(0)}$. Denote $k^{(l)}=\underset{k=1,\dots ,K^{(0)}}{\text{max}}\left|{\mathcal{T}}_{k^{(0)}}^{(0)}\cap {\mathcal{T}}_k^{(l)}\right|$, and we use the model corresponding to ${\mathcal{T}}_{K^{(l)}}^{(l)}$ for prediction. This can be slightly ad hoc, however, literature search does not suggest a more appropriate approach. This procedure is cycled through all subjects, and the prediction MSE values are 0.907 (fusion1), 0.991 (FMR1), 1.130 (respclust1), 1.124 (resiclust1), 1.024 (FMR2), 1.012 (respclust2), 1.081 (resiclust2), and 0.869 (proposed).

As a byproduct of the above procedure, we are also able to evaluate subgrouping stability of the proposed approach. In particular, we compare the heterogeneity structure obtained using all subjects (denoted as φ₀) against that with the lth subject removed (denoted as φ_l), and compute the similarity measure as

$${\left(\begin{array}{c}n-1\\ 2\end{array}\right)}^{-1}\mid \{(i,j):I\left[{\phi }_l\left(x_i,z_i\right)={\phi }_l\left(x_i,z_i\right)\right]=I[{\phi }_0\left(x_i,z_i\right)={\phi }_0\left(x_i,z_i\right)];i<j,i,j\ne l\}\mid ,$$

where l = 1, … , n. Note that this needs to be computed for the subgrouping and sub-subgrouping separately. The mean similarity measure values are 0.843 and 0.913, respectively. The high values provide additional confidence to the proposed analysis. Here we do not pursue the comparison of stability as it may not be sensible to do so. In particular, the “most stable” approach is the one grouping all subjects together and having stability measure 1.

4.2 |. Data-based simulation

Realizing that the parametric-model-based simulation reported in Section 3 may be somewhat simplified, we conduct additional simulation based on the LUAD data. In particular, for each simulation replicate, we randomly select 96 (out of the 118) subjects and adopt the observed imaging feature values. The rest of the simulation settings are similar to those in Section 3. Results are summarized in Figure C15 and Table C20 (Supporting Information). Similar to in Section 3, the proposed approach is observed to have superior performance.

4.3 |. Additional PCA-based analysis

To accommodate the high dimensionality of the second type of imaging features, we have conducted a prescreening. To “complete the picture,” we further consider a dimension reduction-based alternative. In particular, we conduct PCA (principal component analysis) with the second type of imaging features, select the top 10 PCs, and then apply the proposed approach. The estimation results are summarized in Table C21 (Supporting Information). Two subgroups are identified, with sizes 23 and 95, respectively. The second subgroup is further split into four sub-subgroups, with sizes 23, 23, 23, and 26. With the same procedure as described above, the prediction MSE is computed as 0.920. The significant differences between the prescreening and dimension reduction-based analyses are not surprising. Our literature review suggests that there may be more variable selection-based analysis. We leave the systematic comparison between the two types of techniques to future research.

5 |. DISCUSSION

For supervised heterogeneity analysis, which is of critical importance in cancer and other biomedical studies, this study has first proposed an innovative hierarchy. Although limited to the analysis of histopathological imaging features in this article, the hierarchy can have far broader applications. In particular, as long as there is a natural order in the “resolution” of two types of variables (for example, clinical biomarkers + molecular measurements, and main effects + higher order interactions), the hierarchy strategy may be applicable. We have then developed a novel approach to achieve estimation and, more importantly, the hierarchical heterogeneity structure in a single step. The proposed approach has been partly motivated by penalized fusion and sparse group penalization but significantly advances from them. Statistical properties and computation have been carefully studied. It is noted that although having some roots in the existing penalized fusion literature, with the significantly different analysis contexts and penalty form, the developments have been highly nontrivial. Both simulation and data analysis have shown satisfactory performance of the proposed approach. In the analysis of TCGA LUAD data, it has identified a finer heterogeneity structure significantly different from the alternatives.

There are several possible future directions. For example, it is of interest to conduct inference and test whether the proposed hierarchical structure can be simplified or is violated for practical data. This demands a new direction of investigation into inference under penalization. The proposed approach has been designed for low-dimensional variables. Its computational complexity is of the order O(n²(q + p)). The high computational cost problem has been well noted even with the standard penalized fusion (Ma and Huang, 2017; Liu and Lin, 2019; Zhang et al., 2019). It is of interest to extend it to a high-dimensional approach involving additional variable selection or dimension reduction components. Although seemingly “straightforward,” our brief exploration has suggested that tremendous computational and theoretical developments will be needed. It is also of interest to apply the proposed approach to more extensive data analysis. Last but not least, with the lack of direct interpretations of imaging features and lack of independent validation data, we have not been able to fully explore clinical implications of our findings. With the high significance of lung cancer heterogeneity, more careful biological/clinical examination may be warranted.

DATA AVAILABILITY STATEMENT

The data that support the findings in this paper are openly available in TCGA (The Cancer Genome Atlas) at https://portal.gdc.cancer.gov/projects/TCGA-LUAD.