Flexible Bayesian Inference for Identifying Significantly Correlated Multiple Pathway Sets

ABSTRACT

In this paper, we propose a flexible Bayesian inference to identify significantly correlated high‐dimensional functions with the response variable, which is challenging because the relationship between the response variable and high‐dimensional functions is unknown and complex due to the dependence among high‐dimensional functions. For example, in genetics pathway‐based analysis, a pathway is a set of genes that serve a particular cellular or physiological function. A pathway is a high‐dimensional function of genes. A pathway‐based analysis can detect subtle changes in expression levels that are not detectable using a gene‐based analysis. However, these pathways are not independent of each other. Because the clinical outcome is affected by multiple pathway sets, it is inappropriate to model sets using marginal analysis, such as a single‐pathway analysis. Estimating set effects based on a single set ignores the fact that sets interact with each other and, thus, result in false positives or false negatives. In this paper, we propose a generalized fused kernel machine regression to test significantly correlated high‐dimensional functions with the response variable, which can be either continuous or binary variables. We develop a data‐driven, flexible Bayesian inference for adjusting multiple tests using the Bayes factor that accommodates dependence through a simple yet flexible structure. The benefits of our method are illustrated through a simulation study and our motivating data on genetic pathway analysis related to type II diabetes.

1. Introduction

Analyzing correlated high‐dimensional data is a challenging problem in genomics, proteomics, and other related areas. For example, it is important to identify significant genetic pathway effects associated with biomarkers or disease status. A gene pathway is a set of genes that functionally work together to regulate a certain biological process. Pathway‐based analysis can consider the dependency structures among genes and the possibility that several moderately regulated genes may significantly impact the clinical outcomes. As pathways are sets of genes that serve a particular cellular or physiological function, we refer to a pathway as a set and a gene as an element. Pathway‐based analyses have attracted extensive interest [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12] because it is possible to explore multiple pathways with profound impact on human biology and drug development. However, it is challenging to quantify the overall set effect and test which sets are highly associated with the outcome among several multiple pathways due to unknown dependence structures between pathways.

One possible way to estimate the overall set effect is via the kernel machine method, as it is a powerful nonparametric statistical learning model for learning unknown function spaces, especially for high‐dimensional data. A number of methods have been developed for testing the overall set effect [4, 13, 14]. Liu et al. [13] proposed a flexible framework by connecting the kernel machine with a linear fixed model that simultaneously estimates fixed effects and set effects. Liu et al. [13] and Pang et al. [8] developed score tests to identify the overall set effects. This method was extended to predict disease in survival analyses [15, 16].

On the other hand, Stingo et al. [17] and Cheng et al. [2] developed a method to test set effects under the Bayesian framework. Kim et al. [6, 7] considered Bayes factor‐based and resampling‐based methods to identify significant sets. Fang et al. [3, 18] further developed a kernel machine‐based approach for evaluating interactions between pathways and covariates. Fang et al. [18] also developed a nonparametric variable selection procedure for single‐set analysis. Testing for the overall set effect has been well‐studied. However, the standard formulation for estimating the set effect is based on a myopic strategy that assumes only one set at a time. This single‐set‐based test has several limitations. The score test [13, 14] is obtained from an asymptotic distribution of test statistics. The Bayesian survival kernel procedure [16] is developed for a single set analysis. Second, because the outcome is affected by multiple sets, it is inappropriate to model sets using marginal analysis. Estimating set effects based on a single set ignores the fact that sets interact with each other and, thus, result in false positives or false negatives.

Hence, to overcome the limitation of the single‐set‐based test, we propose a fused kernel machine approach to identify significant multiple‐set‐based analyses and develop a data‐driven flexible Bayesian inference to adjust multiple tests using a Bayes factor. We model the unknown high‐dimensional functions of multisets via fused Gaussian kernel machines to consider the possibility that elements within the same set interact and that sets are dependent. This fused structure enables information sharing across related sets, stabilizes inference in high‐dimensional settings, and improves statistical efficiency without requiring explicit knowledge of inter‐set connectivity.

We provide several novel contributions: (1) we develop a generalized fused multi‐kernel machine regression framework that identifies the correlated signal sets associated with continuous or binary response variables; (2) we introduce a Bayesian inference to detect multiple significant set effects simultaneously under a parsimonious dependence structure; and (3) we develop a data‐driven flexible Bayesian inference to adjust multiple tests using the Bayes factor.

The rest of the article is organized as follows. In Section 2, we first describe the problem setup and notation. We introduce the generalized fused kernel machine regression. Section 2.3 explains Bayesian inference to estimate parameters and describe multiple hypothesis tests. Section 3.3 provides how to adjust multiple comparisons using the Bayes factor. In Section 4, we conduct a simulation study to investigate the performance of our methods in terms of true positive rate (TPR), false positive rate (FPR), accuracy, and precision. In Section 5, we demonstrate the advantage of our approach using genetic pathway‐based analysis of type II diabetes [1]. Finally, Section 6 contains concluding remarks.

2. Problem Setup

In this section, we first define some notations, nonparametric models, and test hypotheses.

2.1. Notations and Model Setting

Let $y_i$ be the response variable of the $i\text{th}$ sample, $i=1,\dots ,n$, where $y_i$ can be a continuous or categorical variable. Define $\mathbf{\text{Y}}={(y_1,\dots ,y_i,\dots ,y_n)}^T$ is a $n\times 1$ vector. Let ${\mathbf{\text{z}}}_{i,m}$ be the $n\times p_m$ matrix $p_m$‐element of the $i\text{th}$ sample in the $m\text{th}$ set, $m=1,\dots ,M$, that is, ${\mathbf{\text{z}}}_{i,m}=(z_{1i,m},\dots ,z_{ki,m},\dots ,z_{p_{m}i,m})$. Let ${\mathbf{\text{x}}}_i$ be the $q$‐covariate of the $i\text{th}$ sample, that is, ${\mathbf{\text{x}}}_i=(x_{i1},\dots ,x_{iq})$. Define $\mathbf{\text{Z}}=({\mathbf{\text{Z}}}_1,\dots ,{\mathbf{\text{Z}}}_M)$ is a $n\times {\sum }_{m=1}^M{p}_m$ matrix with ${\mathbf{\text{Z}}}_m={({\mathbf{\text{z}}}_{1,m},\dots ,{\mathbf{\text{z}}}_{n,m})}^T$, which is a $n\times p_m$ matrix, and $\mathbf{\text{X}}={({\mathbf{\text{x}}}_1,\dots ,{\mathbf{\text{x}}}_n)}^T$ is a $n\times q$ matrix.

Let $\mathbf{\text{r}}={({\mathbf{\text{r}}}_1^T,\dots ,{\mathbf{\text{r}}}_m^T,\dots ,{\mathbf{\text{r}}}_M^T)}^T$, where ${\mathbf{\text{r}}}_m^T={(r_{1,m},\dots ,r_{n,m})}^T\sim MN(o,{\tau }_m^2{\mathbf{\text{K}}}_m)$ and $\Vert {\mathbf{\text{r}}}_m{\Vert }_{{\mathbf{\text{K}}}_m}={({\mathbf{\text{r}}}_m^T{\mathbf{\text{K}}}_m^{-1}{\mathbf{\text{r}}}_m)}^{\frac{1}{2}}$, which has the group lasso structure, where ${\mathbf{\text{K}}}_m$ represents the kernel matrix for a $m\text{th}$ set with the $(i,j)\text{th}$ component $k_m({\mathbf{\text{z}}}_{i,m}.{\mathbf{\text{z}}}_{j,m})$. Let $|{\mathbf{\text{r}}}_{m+1}-{\mathbf{\text{r}}}_m{|}_1={\sum }_{i=1}^n|r_{i,m+1}-r_{i,m}|$, which is the $L_1$ norm of the difference between two vectors which has a fused lasso structure.

Rationale for group lasso and fused lasso: Here, a group structure is imposed because the variables within each set act collectively rather than individually. For instance, a gene pathway represents a collection of genes that jointly regulate a specific biological process, and the contribution of individual genes cannot be considered in isolation. The group structure framework accommodates this by specifying a kernel function that models the joint effect of multiple genes within a pathway. This approach allows for nonlinear effects of individual gene expressions and captures the complex interactions that may occur among genes within the same pathway.

We also impose a fused structure to account for the fact that the sets are not independent. Some pathways share genes, while others may not overlap but still display strong correlations through their constituent genes. The fused framework with autocorrelation $AR(1)$ enables us to capture such dependencies among pathways.

We emphasize that the fused‐lasso structure is not intended to represent a biological ordering or a mechanistic sequence among pathways. Instead, it is introduced as a statistical coupling prior across pathway‐specific latent models, with the goal of stabilizing inference and encouraging parsimony in high‐dimensional settings. From a biological perspective, pathways are not isolated entities but overlapping functional modules that often share genes, regulators, and signaling components. Encouraging similarity across pathway‐specific effects is therefore biologically plausible even in the absence of a known or reliable inter‐pathway network. Statistically, the fused‐lasso penalty corresponds to a first‐order Gaussian Markov random field prior and has been widely used as a structured shrinkage and smoothing device [19, 20]. From a modeling standpoint, this structure can be viewed as a path‐graph Laplacian prior, representing one of the simplest forms of graph‐regularized dependence. While more complex graph‐structured priors could in principle be adopted, they would require reliable prior knowledge of inter‐pathway connectivity, which is often unavailable or highly uncertain in practice. Moreover, much of the existing literature [21, 22, 23] on graph‐ or network‐based approaches in pathway genetic analysis has focused primarily on network estimation, rather than on formal hypothesis testing under dependence. Therefore, the proposed fused structure does not aim to recover or represent the true biological pathway network. Rather, it provides a robust and parsimonious dependence structure that enables stable inference and hypothesis testing in the absence of reliable network information. The primary goal of this paper is to develop a flexible Bayesian testing framework under a computationally tractable regularization scheme, rather than to model biological pathway interactions mechanistically. Our proposed Bayesian framework enables hypothesis testing under a simple yet flexible dependence structure.

For example, consider three pathways where pathway 1 shares genes with both pathways 2 and 3, while pathways 2 and 3 do not overlap. This means that the fused lasso component shrinks differences in elements of pathway 1 across neighboring pathway 2 or 3, and smooths adjacent estimates towards one another. Alternatively, if the three pathways are highly correlated or the three pathways have no shared genes but are highly correlated, or the correlation structure among pathways is unknown, all possible distinct orderings of pathways may be considered. The fused structure captures both direct overlaps and indirect correlations between pathways, enabling more accurate detection of their joint association with the response.

Hence, by jointly modeling multiple sets through the group structure and incorporating inter‐pathway correlations through the fused structure, our method improves the accuracy of identifying pathways significantly associated with the response. In contrast, traditional pathway‐based analyses typically adopt a myopic strategy that tests one pathway at a time under an independence assumption, increasing the risk of biased or misleading inference.

2.2. Generalized Fused Multi‐Kernel Machine Regression

By denoting $g(·)$ as a link function, $E(Y)$ as the mean of $Y$, the distribution of $\mathbf{\text{r}}$ given $\mathbf{\text{Z}}$ as $p(·)$, and the conditional prior distribution of $\mathbf{\text{r}}$ as $\pi (·)$, which has the group‐fused structure, we consider the following generalized fused multi‐kernel machine regression model (GFKM):

$$\begin{array}{cc}\hfill & g\{E(\mathbf{\text{Y}})|\mathbf{\text{r}},\mathbf{\text{X}}\}=\mathbf{\text{X}}\mathit{\beta }+\sum _{m=1}^M{\mathbf{\text{r}}}_m\left({\mathbf{\text{Z}}}_m\right);\hfill \\ \hfill & p\left(\mathbf{\text{r}}|\mathbf{\text{Z}},{\sigma }^2,{\tau }_1^2,\dots ,{\tau }_M^2,{\omega }_1^2,\dots ,{\omega }_{M-1}^2\right)\sim N\left[\mathbf{\text{0}},{\sigma }^2{\sum }_{\mathbf{\text{r}}}\left(\mathbf{\text{Z}};{\tau }_1^2,\dots ,{\tau }_M^2,{\omega }_1^2,\dots ,{\omega }_{M-1}^2\right)\right];\hfill \\ \hfill & \pi \left(\mathbf{\text{r}}|\sigma ,{\lambda }_1,{\lambda }_2\right)\propto \exp \left(-\frac{{\lambda }_1}{\sigma }\sum _{m=1}^M\Vert {\mathbf{\text{r}}}_m{\Vert }_{{\mathbf{\text{K}}}_m}-\frac{{\lambda }_2}{\sigma }\sum _{m=1}^{M-1}|{\mathbf{\text{r}}}_{m+1}-{\mathbf{\text{r}}}_m{|}_1\right),\hfill \end{array}$$ (1)

where $\mathit{\beta }$ is a $q\times 1$ vector of regression coefficients for the covariate effects, ${\mathbf{\text{r}}}_m({\mathbf{\text{Z}}}_m)$ is an unknown function in reproducing kernel Hilbert space (RKHS), variance ${\sigma }^2$ is an unknown parameter of $\mathbf{\text{Y}}$ for a continuous response but ${\sigma }^2=1$ for binary response with probit link, parameters ${\mathit{\tau }}^2=({\tau }_1^2,{\tau }_2^2,\dots ,{\tau }_m^2)$ are associated with group lasso, and parameters ${\mathit{\omega }}^2=({\omega }_1^2,{\omega }_2^2,\dots ,{\omega }_{m-1}^2)$ are associated with a fused lasso, both of which parameterize ${\sum }_{\mathbf{\text{r}}}$, that is, ${\sum }_{\mathbf{\text{r}}}(\mathbf{\text{Z}};{\tau }_1^2,\dots ,{\tau }_M^2,{\omega }_1^2,\dots ,{\omega }_{M-1}^2)$, ${\tau }_m^2$ is controlled by ${\lambda }_1^2$, and ${\omega }_m^2$ is controlled by the ${\lambda }_2^2$. Note that ${\lambda }_1^2$ is the tuning parameter for the group lasso, and ${\lambda }_2^2$ is the tuning parameter for the fused lasso.

We simply denote ${\sum }_{\mathbf{\text{r}}}(\mathbf{\text{Z}};{\tau }_1^2,\dots ,{\tau }_M^2,{\omega }_1^2,\dots ,{\omega }_{M-1}^2)$ as ${\sum }_{\mathbf{\text{r}}}$. The form of ${\sum }_{\mathbf{\text{r}}}^{-1}$ is derived from representing the Laplace (double exponential) conditional prior of $\mathbf{\text{r}}|{\sigma }^2,{\lambda }_1,{\lambda }_2$ as a scale mixture of a normal distribution combined with an exponential density [24]. The complete form of ${\sum }_{\mathbf{\text{r}}}^{-1}$ can be expressed as follows:

$${\sum }_{\mathbf{\text{r}}}^{-1}={\left(\begin{array}{lllll}{\sum }_{11}^{-1}& {\sum }_{12}^{-1}& \mathbf{\text{0}}& \cdots & \mathbf{\text{0}}\\ {\sum }_{21}^{-1}& {\sum }_{22}^{-1}& \ddots & \ddots & ⋮\\ \mathbf{\text{0}}& \ddots & \ddots & \ddots & \mathbf{\text{0}}\\ ⋮& \ddots & \ddots & \ddots & {\sum }_{(M-1)M}^{-1}\\ \mathbf{\text{0}}& \cdots & \mathbf{\text{0}}& {\sum }_{M(M-1)}^{-1}& {\sum }_{MM}^{-1}\end{array}\right)}_{,}$$

where 0 is a $n\times n$ matrix with all zero elements and ${\sum }_{(m-1)m}^{-1}$ and ${\sum }_{m(m-1)}^{-1}$ are block matrices of size $n\times n$.

This matrix ${\sum }_{\mathbf{\text{r}}}^{-1}$ depends on parameters ${\mathit{\tau }}^2=({\tau }_1^2,{\tau }_2^2,\dots ,{\tau }_M^2)$, ${\mathit{\omega }}^2=({\omega }_1^2,{\omega }_2^2,\dots ,{\omega }_{M-1}^2)$, and ${\mathbf{\text{K}}}_m({\mathbf{\text{Z}}}_m)$. The form of ${\sum }_{mm}^{-1}$, ${\sum }_{(m-1)m}^{-1}$ and ${\sum }_{m(m-1)}^{-1}$ are

• ${\sum }_{mm}^{-1}=1/{\tau }_m^2{{\mathbf{\text{K}}}_m}^{-1}+1/{\omega }_m^2{\mathbf{\text{I}}}_n$ for $m=1,M$,

• ${\sum }_{mm}^{-1}=1/{\tau }_m^2{{\mathbf{\text{K}}}_m}^{-1}+(1/{\omega }_{m-1}^2+1/{\omega }_m^2){\mathbf{\text{I}}}_n$ for $m=2,\dots ,M-1$,

• ${\sum }_{(m-1)m}^{-1}={\sum }_{m(m-1)}^{-1}=1/{\omega }_{m-1}^2{\mathbf{\text{I}}}_n$ for $m=2,\dots ,M$,

where the kernel structure assigned to each ${\mathbf{\text{r}}}_m$ leads to the diagonal block of size $n\times n$, while the off‐diagonal components involving the ${\omega }_m$ parameters work to shrink random effects that are adjacent sets. We consider that the prior distribution for ${\tau }_m^2$ is controlled by ${\lambda }_1^2$, and ${\omega }_m^2$'s prior distribution is controlled by ${\lambda }_2^2$. Thus, ${\lambda }_1^2$ is the tuning parameter for the group prior, and ${\lambda }_2^2$ is the tuning parameter for the fused prior. The detailed prior specification is in Section 2.3.

2.3. Bayesian Sampling for Parameter Estimation

This section describes how to estimate parameters in GFKM (1). We explain the prior specification and illustrate the full conditional distribution for continuous and binary variable cases. For the binary response variable, we use the probit link function. The specification of prior distributions is as follows:

$$\begin{array}{ccc}\hfill & p(\mathit{\beta })\sim N\left({\mu }_{\beta },{\sigma }_{\beta }^2\right);{\sigma }^2\sim \left\{\begin{array}{ll}\text{IG}(\mu ,\nu )& \text{if y is continuous}\\ 1& \text{if y is binary}\end{array}\right.\hfill & \hfill \\ \hfill & \mathbf{\text{r}}|{\sigma }^2,{\tau }_1^2,\dots ,{\tau }_M^2,{\omega }_1^2,\dots ,{\omega }_{M-1}^2\sim N\left(\mathbf{\text{0}},{\sigma }^2{\sum }_{\mathbf{\text{r}}}\right);\hfill & \\ \hfill & {\tau }_1^2,\dots ,{\tau }_M^2\sim \text{Gamma}\left(\frac{n+1}{2},\frac{{\lambda }_1^2}{2}\right);{\omega }_1^2,\dots ,{\omega }_{M-1}^2\sim \text{Gamma}\left(1,\frac{{\lambda }_2^2}{2}\right);\hfill & \\ \hfill & {\lambda }_h^2\sim \text{Gamma}\left({\gamma }_h,{\delta }_h\right)\text{for}h=1,2;{\rho }_m\sim \text{Gamma}\left(a_m,g_m\right),\hfill & \end{array}$$ (2)

where the hyperparameters (${\mu }_{\beta }$, ${\sigma }_{\beta }$) for the prior on the regression coefficients ($\beta$) are set to ($1,0.1^2$), $\text{IG}(\mu ,\nu )$ is inverse gamma distribution with shape parameter $\mu$ and scale parameter $\nu$. The full form of ${\sum }_{\mathbf{\text{r}}}^{-1}$ is described in Section 2.1, respectively. The full conditional distributions for all parameters except for ${\rho }_m$ have closed forms. So, we draw the MCMC sample using Metropolis‐Hastings (MH) for ${\rho }_m$ and Gibbs sampling for all parameters except for ${\rho }_m$. The detailed forms of the conditional distributions for continuous and binary responses are summarized in Appendix A.

The rationale for defining the regularization parameters as ${\lambda }_1^2$ and ${\lambda }_2^2$ is as follows. The conditional prior $\mathbf{\text{r}}|{\sigma }^2$ reflects the Bayesian group and fused lasso formulation introduced in Section 2.2, Equation (1). This conditional prior can be expressed as a mixture of Gaussians, as shown in the two equations described in Section S1 of Supporting Information. The squared form of ${\lambda }_1^2$ appears naturally in the parameters of the Gamma distribution. By defining the regularization parameter in squared form, the resulting integral becomes tractable and yields a closed‐form expression. In this setup, the Gamma distribution acts as a mixing distribution for the variance term ($1/{\tau }_m^2$) of the Gaussian. The same reasoning applies to the parameters ${\lambda }_2^2$ and ${\omega }_m^2$, which play an analogous role. Hyperparameters are selected via grid search, and the optimal values are those that maximize the likelihood function.

3. Hypothesis Test for Identifying Correlated Multiple Sets of Variables

Our main question of interest is to identify correlated multiple sets of variables associated with the response. We conduct the statistical inference based on the Bayes factor [25].

To identify which set is associated with the response, one may consider the following null and alternative hypothesis, $H_{0,m}$: $r_m({\mathbf{\text{Z}}}_m)=0$ vs. $H_{1,m}$: $r_m({\mathbf{\text{Z}}}_m)\ne 0$. Let $B{F}_{10}$ denote the Bayes factor (BF) in favor of alternative hypothesis $H_{1,m}$ against null hypothesis $H_{0,m}$. That is,

$$B{F}_{10}=\frac{p\left(\mathbf{\text{y}}|H_{1,m}\right)}{p\left(\mathbf{\text{y}}|H_{0,m}\right)}=\frac{\int p\left(\mathbf{\text{y}}|{\mathbf{\Omega }}_1,H_{1,m}\right)p\left({\mathbf{\Omega }}_1|H_{1,m}\right)d{\mathbf{\Omega }}_1}{\int p\left(\mathbf{\text{y}}|{\mathbf{\Omega }}_0,H_{0,m}\right)p\left({\mathbf{\Omega }}_0|H_{0,m}\right)d{\mathbf{\Omega }}_0},$$

where $p(\mathbf{\text{y}}|H_{l,m})$ is the marginal likelihood of data y under $H_{l,m}$ ($l=0$ or 1) given the model‐specific parameter vector ${\mathbf{\Omega }}_l=(\mathit{\beta },{\sigma }^2,{\mathit{\tau }}^2,{\mathit{\omega }}^2,{\lambda }_1^2,{\lambda }_2^2)$, $l=0,1$. $p(\mathbf{\text{y}}|{\mathbf{\Omega }}_l,H_{l,m})$ is the density function under $H_{l,m}$ given ${\mathbf{\Omega }}_l$, and $p({\mathbf{\Omega }}_l|H_{l,m})$ is the prior distribution density under $H_{l,m}$. Because the integral for marginal likelihood under $H_{l,m}$ does not have a closed form, we approximate this integration using the following method proposed by Newton and Raftery [26],

$$\widehat{p}\left(\mathbf{\text{y}}|H_{l,m}\right)={\left[\frac{1}{S}\sum _{s=1}^S\left\{\frac{1}{p\left(\mathbf{\text{y}}|{{\mathbf{\Omega }}_l}^s,H_{l,m}\right)}\right\}\right]}^{-1},$$

where ${({\mathbf{\Omega }}_l^1,\dots ,{\mathbf{\Omega }}_l^S)}^T$ are MCMC draws of size $S$ from the posterior distribution $p({\mathbf{\Omega }}_l|\mathit{y},H_{l,m})$. Larger values of $B{F}_{10}$ suggest that the data favor the model under the alternative hypothesis $H_{1,m}$. Large values of BF favor $H_{1,m}$. This means that the data indicate that $H_{1,m}$ is more strongly supported by the data than $H_{0,m}$. Kass and Rafter [25] suggested how to interpret the value of BF. We interpret the value of BF as not worth more than a bare mention if $1<BF\le 3$, positive if $3<BF\le 20$, strong if $20<BF\le 150$, and very strong if $BF>150$. However, this hypothesis is formulated under the assumption that $r_m(·)$ functions are independent, and the interpretation of $BF$ does not incorporate adjustments for multiple comparisons. Thus, we suggest a data‐driven flexible Bayesian inference based on $BF$ to adjust multiple tests in Section 3.3.

3.1. Multiple Hypothesis Tests

As we specified in GFKM(1), the set ${\mathbf{\text{r}}}_m$ depends on ${\tau }_m$, ${\omega }_m$, or both ${\omega }_{m-1}$ and ${\omega }_m$, where ${\omega }_m$ shrinks random effects of ${\mathbf{\text{r}}}_{m-1}$ and ${\mathbf{\text{r}}}_m$. The ${\tau }_m$ is used to evaluate the existence of the overall set effect associated with the response variable. We also use ${\omega }_m$ or both ${\omega }_{m-1}$ and ${\omega }_m$ to evaluate the existence of fused kernel structures between sets associated with the response variable. As a simple illustration, we consider $M=3$. When the number of sets is three, the ${\mathbf{\text{r}}}_1$ depends on the overall set effect ${\tau }_1$ and ${\omega }_1$. The ${\mathbf{\text{r}}}_2$ has ${\tau }_2$, ${\omega }_1$, and ${\omega }_2$ since the set ${\mathbf{\text{r}}}_2$ has a correlation structure with ${\mathbf{\text{r}}}_1$ and ${\mathbf{\text{r}}}_3$. The ${\mathbf{\text{r}}}_3$ has ${\tau }_3$, and ${\omega }_2$.

3.2. Null and Alternative Hypotheses

The corresponding null and alternative hypothesis are denoted as $H_{0,\ell }$ and $H_{1,\ell }$, $\ell =1,2,3$:

• $H_{0,1}$ is ${\mathbf{\text{r}}}_1=0$, which is the same as ${\tau }_1={\omega }_1=0$, and $H_{1,1}$ is ${\mathbf{\text{r}}}_1\ne 0$, which is the same as ${\tau }_1,{\omega }_1>0$.

• $H_{0,2}$ is ${\mathbf{\text{r}}}_2=0$, which is the same as ${\tau }_2={\omega }_1={\omega }_2=0$, and $H_{1,2}$ is ${\mathbf{\text{r}}}_2\ne 0$, which is the same as ${\tau }_2,{\omega }_1,{\omega }_2>0$.

• $H_{0,3}$ is ${\mathbf{\text{r}}}_3=0$, which is the same as ${\tau }_3={\omega }_3=0$, and $H_{1,3}$ is ${\mathbf{\text{r}}}_3\ne 0$, which is the same as ${\tau }_3,{\omega }_2>0$.

Using three sets, we can think of $3!$ ordering of three sets. However, the model estimations of GFKM using some of the orderings of the three sets are the same. For example, the GFKM estimation based on the ordering (${\mathbf{r}}_1$, ${\mathbf{r}}_2$, ${\mathbf{r}}_3$) is the identical to that based on the reverse ordering (${\mathbf{r}}_3$, ${\mathbf{r}}_2$, ${\mathbf{r}}_1$), because the correlation structure between ${\mathbf{r}}_1$ and ${\mathbf{r}}_2$ is the same as that between ${\mathbf{r}}_2$ and ${\mathbf{r}}_1$. Also, the model estimation using the following order (${\mathbf{r}}_1$, ${\mathbf{r}}_3$, ${\mathbf{r}}_2$) is also same as that of (${\mathbf{r}}_2$, ${\mathbf{r}}_3$, ${\mathbf{r}}_1$). Similarly, we can have the same model estimation between the models using the order (${\mathbf{r}}_2$, ${\mathbf{r}}_1$, ${\mathbf{r}}_3$) and the order (${\mathbf{r}}_3$, ${\mathbf{r}}_1$, ${\mathbf{r}}_2$). Figure 1 shows which orderings of three sets are identical or not. Hence, we only need to consider three models: (1) GFKM with (${\mathbf{r}}_1$, ${\mathbf{r}}_2$, ${\mathbf{r}}_3$); (2) GFKM with (${\mathbf{r}}_1$, ${\mathbf{r}}_3$, ${\mathbf{r}}_2$); and (3) GFKM with (${\mathbf{r}}_2$, ${\mathbf{r}}_1$, ${\mathbf{r}}_3$). For each model, we conduct the following hypothesis test: $H_{0,\ell }$ versus $H_{1,\ell }$, $\ell =1,2,3$.

FIGURE 1.

The orderings of three sets. The GFKM with the following order of sets (${\mathbf{r}}_1$,${\mathbf{r}}_2$,${\mathbf{r}}_3$) is the same as that using the order (${\mathbf{r}}_3$,${\mathbf{r}}_2$,${\mathbf{r}}_1$); The GFKM with the following order of sets (${\mathbf{r}}_1$,${\mathbf{r}}_3$,${\mathbf{r}}_2$) is equal to that of (${\mathbf{r}}_2$,${\mathbf{r}}_3$,${\mathbf{r}}_1$); the GFKM based on the following ordering sets (${\mathbf{r}}_2$,${\mathbf{r}}_1$,${\mathbf{r}}_3$) matches that of ordering (${\mathbf{r}}_3$,${\mathbf{r}}_1$,${\mathbf{r}}_2$).

Using our approach, we can identify which three pathways are associated with the response variable by accounting for the possibility that one pathway may be strongly connected to the other two, through the execution of three distinct tests. These connections reflect inter‐pathway relationships that cannot be detected under the assumption of pathway independence. Therefore, our method captures both direct overlaps and indirect correlations among pathways, enabling more accurate detection of their joint association with the response.

3.3. Data‐Driven Adjustment of Bayes Factor for Multiple Tests

The multiplicity arises when multiple hypothesis tests are conducted, leading to an increased likelihood of false positives. In the sets, failing to account for multiplicity can lead to incorrect identification of sets because multiple sets have overlapping elements and element interactions with other sets. Thus, the decision of cutoff values for BF to account for multiplicity is essential for reliable set identification. Traditional Bayesian approaches decide an arbitrary threshold for BF, such as 1, 3, 20, 150 [25] to determine the strength of evidence. However, this approach may not consider the effect of multiplicity. Zhang et al. [16] suggested a method to determine a threshold of BF for multiple comparisons under a single set test. We extend this method to adjust BF for the fused multi‐kernel sets test that accounts for the complexity of multiplicity, thereby providing more reliable criteria for evaluating BF using a data‐driven procedure.

Consider $C\times M$ sets with $C$ combinations and $M$ sets in the model. We built GFKM on $M$ sets with $M$. Consider hypotheses tests $H_{0,m}$ and $H_{1,m}$, $m=1,\dots ,M$. Let $\mathbf{\text{BF}}=(B{F}_1^1,\dots ,B{F}_1^M,\dots ,B{F}_c^m,\dots ,B{F}_C^1,\dots ,B{F}_C^M)$, where $B{F}_c^m,$ is the Bayes factor for the $c_m$‐$th$ set ($c=1,\dots ,C$, $m=1,\dots ,M$). For the $c_m$‐$th$ set, $B{F}_c^m\mathrm{s}$ are computed multiple times (e.g., 100, 500, 1000 times) with different MCMC samples of the same sample size and burn‐in. The final ${\widehat{BF}}_c^m$ is determined by averaging the $B{F}_c^m\mathrm{s}$ that fall within the first and third quantiles of the collected $B{F}_c^m\mathrm{s}$. We remove outliers for $\widehat{\mathbf{\text{BF}}}=({\widehat{BF}}_1^1,\dots ,{\widehat{BF}}_1^M,\dots ,{\widehat{BF}}_C^M)$ because they highly affect the nonsignificant decision. Outliers of ${\widehat{BF}}_c^{mc}$ were defined as those with Bayes factor values greater than 100 000. These removed outliers were directly interpreted as significant results. Since the Bayes factor of this magnitude provides overwhelming evidence in favor of the alternative hypothesis, no additional multiple comparison adjustment was applied to them.

Let $G$ be an arbitrary number of classes ($G\le C\times M$), and $\mu =({\mu }_1,\dots ,{\mu }_G)$ represents a vector of class centers. Define $\zeta =({\zeta }_{1_{1}g},\dots ,{\zeta }_{1_{M}g},\dots ,{\zeta }_{C_{M}G})$ as a label vector $c_m$‐$th$ set ($c=1,\dots ,C$, $m=1,\dots ,M$), where the ${\zeta }_{c_{m}g}=1$ if $c_m$‐$th$ set belongs to the $c_m$‐$th$ class, and ${\zeta }_{c_{m}v}=0$ for all $v\ne g$. To decide the number of classes for BF (called $G^{\ast }$), the steps of our algorithm are summarized as follows:

• Step 1: $\widehat{\mathbf{\text{BF}}}$ standardization. We standardize ${\widehat{BF}}_c^m$ ($c=1,\dots ,C$, $m=1,\dots ,M$) by using following formula,

  $${\widetilde{BF}}_c^m=\frac{{\widehat{BF}}_c^m-\overline{BF}}{S{D}_{\widehat{\mathbf{\text{BF}}}}},$$

  where $\overline{BF}$ and $S{D}_{\widehat{\mathbf{\text{BF}}}}$ are mean and standard deviation of $\widehat{\mathbf{\text{BF}}}$, respectively.

• Step 2: Determine the objective function. To calculate the distance or similarity of two points, we use Euclidean distance. To minimize variance within class, our objective function is $Q={\text{argmin}}_{{\mu }_g}{\sum }_{c=1}^C{\sum }_{m=1}^M{\sum }_{g=1}^G||{\widetilde{BF}}_c^m-{\mu }_g|{|}^2$.

• Step 3: Optimize the objective function. First, we candidate initial values $\mu =({\mu }_1,\dots ,{\mu }_G)$ of each class ($G\le CM$). Second, update $\zeta$ using objective function ($Q$), which means that assign each ${\widetilde{BF}}_c^m$ to nearest initial value ${\mu }_g$ (assign $c_m$‐$th$ set to $g$‐$th$ class). Third, we update each ${\mu }_g$ such that ${\mu }_g={\sum }_{c=1}^C{\sum }_{m=1}^M{\zeta }_{c_{m}g}{\widetilde{BF}}_c^m/{\sum }_{c=1}^C{\sum }_{m=1}^M{\zeta }_{c_{m}g}$. Lastly, update $\zeta$ and $\mu$ until ${\sum }_{g=1}^G||{\mu }_g^{(iter+1)}-{\mu }_g^{(iter)}|{|}^2\le 10^{-5}$, where ${\mu }_g^{(iter)}$ is previous value of ${\mu }_g^{(iter+1)}$ for iterations, and $10^{-5}$ is criterion, which can be smaller values.

We get the range of $B{F}_{g,adj}$ ($U{B}_g<B{F}_{g,adj}<L{B}_{g+1}$), $g=1,2,3$, where $L{B}_{g+1}$ is the lower bound of $g+1\text{th}$ class and $U{B}_g$ is the upper bound of the $g\text{th}$ class. We decide on cutoff values for BF within any values in the $B{F}_{g,adj}$ range. When determining the number of classes for BFs $G^{\ast }$, there are four cases as follows:

1. When $G^{\ast }=2$, if $B{F}_c^m$ is included in the higher class, then the $c_m$‐$th$ set is considered to be a significant set ($B{F}_c^m\ge B{F}_{1,adj}$), whereas if $B{F}_c^m$ is included in the lower class, then the $c_m$‐$th$ set is considered to be a insignificant set. To establish the cutoff values for determining a significant set and an insignificant set, use the range within which $B{F}_{1,adj}$ falls, specifically $[U{B}_1,L{B}_2]$.

2. When $G^{\ast }=3$, if $B{F}_c^m$ is close to the center of the highest class, then the $c_m$‐$th$ set is regarded to be a strongly significant set ($B{F}_c^m\ge B{F}_{2,adj}$). If $B{F}_c^m$ belongs to the second highest class, then the $c_m$‐$th$ set is considered to be a significant set ($B{F}_c^m\ge B{F}_{1,adj}$). To determine cutoff values for identifying a strongly significant set and a significant set, use the range in which $B{F}_{2,adj}$ lies, namely $[U{B}_2,L{B}_3]$. The $c_m$‐$th$ set is considered to be a insignificant set, when $B{F}_c^m$ is close to the center of the last class. To establish the cutoff values for determining a significant set and an insignificant set, use the range within which $B{F}_{1,adj}$ falls, specifically $[U{B}_1,L{B}_2]$.

3. When $G^{\ast }=4$, if $B{F}_c^m$ is neighboring the center of the highest class, then the $c_m$‐$th$ set is considered to be a very strongly significant set. We consider the $c_m$‐$th$ set to be a strongly significant set with $B{F}_c^m$ in the second highest class. To set cutoff values for classifying a very strongly significant set ($B{F}_c^m\ge B{F}_{3,adj}$) and a strongly significant set ($B{F}_c^m\ge B{F}_{2,adj}$), use the range within which $B{F}_{3,adj}$ falls, specifically $[U{B}_3,L{B}_4]$. When $B{F}_c^m$ is involved in the third‐highest class, the $c_m$‐$th$ set is regarded to be a significant set ($B{F}_c^m\ge B{F}_{1,adj}$). To determine cutoff values for identifying a strongly significant set and a significant set, use the range in which $B{F}_{2,adj}$ lies, namely $[U{B}_2,L{B}_3]$. The $c_m$‐$th$ set is considered to be a insignificant set, when $B{F}_c^m$ is adjacent to the center of the last class. To establish the cutoff values for determining a significant set and an insignificant set, use the range within which $B{F}_{1,adj}$ falls, specifically $[U{B}_1,L{B}_2]$.

The entire procedure of a data‐driven adjustment of the BF‐based test (denoted adj‐BF) for multiple hypothesis tests is summarized in Algorithm 1, and the flowchart of adj‐BF is displayed in Figure 2.

ALGORITHM 1. Data‐driven adjustment of BF method for multiple hypothesis tests (adj‐BF).

FIGURE 2.

The flowchart of adjustment of BF‐based test (adj‐BF) for multiple hypothesis tests: $B{F}_{g,adj}$ can be any value in the range [$U{B}_g$, $L{B}_{g+1}$], $g=1,2,3$.

4. Simulation

We conducted a simulation study to investigate the performance of our approach. Our test is a BF based on a GFKM, denoted as BF‐GFKM.

There are two comparison methods. One is a frequent test based on a semiparametric additive kernel machine regression (FT‐SAKM) [27], which is a random effect model‐based approach for continuous response data that jointly considers multiple kernel functions. FT‐SAKM's test is based on the score test. The other is a BF based on the generalized additive kernel machine model (BF‐GAKM), in which we could build our model (1) with ${\lambda }_2=0$. BF‐GAKM can be the Bayesian version of Schweiger et al. [27].

When the response variable is continuous, we compared our BF‐GFKM with FT‐SAKM and BF‐GAKM. We also compare them using the adj‐BF threshold with those using traditional BF cut thresholds such as 1, 3, 5, and 10 for a continuous response. For the binary response variable, we compared our BF‐GFKM with BF‐GAKM. We also compare them using the adj‐BF threshold with those using traditional BF cut thresholds such as 1, 1.5, and 2 for binary response cases.

We considered three simulation scenarios: (i) correlated sets without shared elements, (ii) independent sets without shared elements, and (iii) correlated sets with shared elements. In addition, we conducted an additional simulation study under a more complex strong dependence structure than AR(1), which is summarized in Section S3.4 of the Supporting Information. The simulated dependence structure in S3.4 does not satisfy the defining properties of an AR(1) model, as correlations are neither determined by a single decay parameter nor monotone in index distance, and do not rely on an ordering of locations. The correlation structure represents a meaningful misspecification relative to the assumed model. We also performed a sensitivity analysis by varying the hyperparameters of the regularization terms, as summarized in Section S4 of the Supporting Information.

We compared the performance of our approach with the alternative in terms of TPR, FPR, accuracy, and precision. TP, FP, FN, and TN are defined as follows:

$$\begin{array}{cc}\hfill TP& =\sum _{s=1}^{100}\mathbb{1}\left({\widehat{BF}}_s>B{F}_{1,adj},{\widehat{BF}}_s>B{F}_{1,adj}\right),\hfill \\ \hfill FP& =\sum _{s=1}^{100}\mathbb{1}\left({\widehat{BF}}_s<B{F}_{1,adj},{\widehat{BF}}_s>B{F}_{1,adj}\right),\hfill \\ \hfill FN& =\sum _{t=1}^{100}\mathbb{1}\left({\widehat{BF}}_s>B{F}_{1,adj},{\widehat{BF}}_s<B{F}_{1,adj}\right),\hfill \\ \hfill TN& =\sum _{t=1}^{100}\mathbb{1}\left({\widehat{BF}}_s<B{F}_{1,adj},{\widehat{BF}}_s<B{F}_{1,adj}\right).\hfill \end{array}$$

Evaluation metrics, which are TR rate (TPR), FP rate (FPR), Accuracy, and Precision, are defined as follows,

$$\begin{array}{cc}\hfill TPR& ={\frac{TP}{TP+FN}}_{,}FPR={\frac{FP}{TN+FP}}_{,}\hfill \\ \hfill Accuracy& ={\frac{TP+TN}{TP+FP+TN+FN}}_{,}Precision={\frac{TP}{TP+FP}}_{.}\hfill \end{array}$$

These criteria are evaluated by testing hypotheses using adj‐BF. We considered cutoff values for the significant set and the insignificant set. The significant sets are estimated BF greater than cutoff values at $s\text{th}$ simulation (${\widehat{BF}}_s>B{F}_{1,adj}$), where $B{F}_{1,adj}$ is denoted as the cutoff value of the BF obtained from our adj‐BF method for multiple testings, and $B{F}_{1,adj}$ is any value in the range [$U{B}_1$, $L{B}_1$].

4.1. Correlated Sets Without Shared Elements

We considered four simulation settings for continuous response, as well as two simulation settings for binary response in Section 4.1.1. For each case, we conducted three hypotheses ($H_{0,m}$ and $H_{1,m}$, $m=1,2,3$) when $M=3$. We compared the performance of our approach with the alternative in terms of TPR, FPR, accuracy, and precision.

4.1.1. Setting

We set $M=3$ (three sets) and $q=2$ (two covariates). Because each unknown function has a different number of elements, we set $\mathbf{p}=(p_1,\dots ,p_M)$, which is a vector of the different numbers of elements for each unknown function. We varied the values of $n$ and p to represent the situations in which $p_m>n$ for $m=1,\dots ,M$. The simulation settings 1–2 are for continuous response, and 3 is for binary response. For each combination of $(n,\mathbf{p})$, three different hypothesis tests were conducted, and 100 simulations were run. We ran 10 000 MCMC iterations with 2000 burn‐in times. We kept one sample for every five draws to reduce autocorrelation in MCMC samples. The computational complexity for fitting our model is $O({(nM)}^3)$ and the computational time for the hypothesis across different simulation cases and sample sizes is summarized in S2 of the Supporting Information.

We first generated the covariates ${\mathbf{\text{x}}}_i={(x_{i1},x_{i2})}^T,x_{i1}\sim N(10,1),x_{i2}\sim Ber(0.5)$, and set ${\mathit{\beta }}_{true}=(1,1)$. We considered $r_m({\mathbf{z}}_{\mathbf{i}\mathbf{,}\mathbf{m}})=0.5\{r_{common}({\mathbf{z}}_{\mathbf{i}\mathbf{,}\mathbf{m}})+r_m^{\ast }({\mathbf{z}}_{\mathbf{i}\mathbf{,}\mathbf{m}})\}$, $m=1,2,3$, which contains two parts: shared common ($r_{commom}$) and nonshared ($r_m^{\ast }$) parts because unknown functions have shared elements and nonshared elements. Because $r_m({\mathbf{z}}_{\mathbf{i}\mathbf{,}\mathbf{m}})$ can be varied by an unknown or misspecified mechanism, $r_m({\mathbf{z}}_{\mathbf{i}\mathbf{,}\mathbf{m}})$ is differently defined, as described in Cases 1–3, which are explained in brief. We also generated $\mathbf{\text{Z}}=({\mathbf{\text{Z}}}_1,\dots ,{\mathbf{\text{Z}}}_M)$ from $N(\mathbf{\text{0}},\sum )$ with the correlation matrix $\sum =A⨂B$, where $A=corr({\mathbf{\text{z}}}_{i,m},{\mathbf{\text{z}}}_{j,m})=0.2{\mathbf{\text{I}}}_n+0.8J_M$, $J_M$ is a $M\times M$ matrix with all‐ones components, and

$$B=corr({\mathbf{\text{Z}}}_l,{\mathbf{\text{Z}}}_m)=\left(\begin{array}{lll}1& 0.750& 0.563\\ 0.750& 1& 0.750\\ 0.563& 0.750& 1\end{array}\right)$$

which has a AR(1) correlation for $l,m=1,2,3$. Each $z_{ki,m}$ was generated from N(0,1), $(i=1,\dots ,n;k=1,\dots ,p_m;m=1,2,3)$ and then $z_{ki,m}$ were transformed to Uniform(−2,2). A more complex dependence structure than AR(1) is considered in Section S3.4.

To generate a continuous response variable, we considered the following Cases 1–2 with $(n,\mathbf{\text{p}})=(n,p_1,p_2,p_3)\in \{(50,60,55,50),(100,110,105,100\}$. To generate a binary response variable, we used the probit link and considered Case 3 with $(n,\mathbf{\text{p}})=(n,p_1,p_2,p_3)\in \{(80,90,85,80),(100,110,105,100)\}$:

• Case 1: $y_i={\mathbf{\text{x}}}_i^T\mathit{\beta }+{\sum }_{m=1}^3{r}_m({\mathbf{\text{z}}}_{i,m})+{ϵ}_i$ with ${ϵ}_i\sim N(0,1)$, $r_m({\mathbf{z}}_{\mathbf{i}\mathbf{,}\mathbf{m}})=0.5\{r_{common}({\mathbf{z}}_{\mathbf{i}\mathbf{,}\mathbf{m}})+r_m^{\ast }({\mathbf{z}}_{\mathbf{i}\mathbf{,}\mathbf{m}})\}$, where $r_{common}({\mathbf{z}}_{\mathbf{i}\mathbf{,}\mathbf{m}})=z_{1i,m}^3-z_{2i,m}^2+z_{1i,m}+z_{2i,m}$, ${\mathbf{\text{z}}}_{i,m}={(z_{1i,m},\dots ,z_{p_{m}i,m})}^T$, $r_1^{\ast }({\mathbf{z}}_{\mathbf{i}\mathbf{,}\mathbf{1}})=-0.5exp(z_{3i,1})cos(z_{4i,1})+z_{5i,1}-z_{6i,1}+sin(z_{7i,1})+cos(z_{8i,m})+0.5z_{9i,1}{z}_{10i,1}$, $r_2^{\ast }({\mathbf{z}}_{\mathbf{i}\mathbf{,}\mathbf{2}})=-0.5exp(z_{3i,2})cos(z_{4i,2})+sin(z_{5i,2})-z_{6i,2}+sin(z_{7i,2})+cos{(z_{8i,2})}^2+0.5z_{9i,2}{z}_{10i,2}+cos{(z_{11i,2})}^2+0.5z_{12i,2}{z}_{13i,2}+cos{(z_{14i,2})}^2$, and $r_3^{\ast }({\mathbf{z}}_{\mathbf{i}\mathbf{,}\mathbf{3}})=-0.5exp(z_{3i,3})z_{4i,3}+sin{(z_{6i,3})}^2+0.5sin(z_{4i,3})cos(z_{5i,3})+z_{7i,3}^2+z_{7i,3}$.

• Case 2: $y_i={\mathbf{\text{x}}}_i^T\mathit{\beta }+{\sum }_{m=1}^3{r}_m({\mathbf{\text{z}}}_{i,m})+{ϵ}_i$, $r_m({\mathbf{z}}_{\mathbf{i}\mathbf{,}\mathbf{m}})=0.35\{r_{common}({\mathbf{z}}_{\mathbf{i}\mathbf{,}\mathbf{m}})+r_m^{\ast }({\mathbf{z}}_{\mathbf{i}\mathbf{,}\mathbf{m}})+1\}$, where $r_{common}({\mathbf{z}}_{\mathbf{i}\mathbf{,}\mathbf{m}})=z_{1i,m}^2+z_{2i,m}^2+z_{3i,m}^2+z_{4i,m}^2+z_{5i,m}^2$, ${\mathbf{\text{z}}}_{i,m}={(z_{1i,m},\dots ,z_{p_{m}i,m})}^T$, $r_1^{\ast }({\mathbf{z}}_{\mathbf{i}\mathbf{,}\mathbf{1}})=0$, $r_2^{\ast }({\mathbf{z}}_{\mathbf{i}\mathbf{,}\mathbf{2}})=z_{6i,2}^2+z_{7i,2}^2$, and $r_3^{\ast }({\mathbf{z}}_{\mathbf{i}\mathbf{,}\mathbf{3}})=z_{6i,3}^2$.

• Case 3: $y_i=1$ if $y_i^{\ast }\ge 0$ or $y_i=0$ otherwise with $y_i^{\ast }={\mathbf{\text{x}}}_i^T\mathit{\beta }+{\sum }_{j=1}^3{r}_j({\mathbf{\text{z}}}_{i,j})$, $r_m({\mathbf{z}}_{\mathbf{i}\mathbf{,}\mathbf{m}})=0.2\{r_{common}({\mathbf{z}}_{\mathbf{i}\mathbf{,}\mathbf{m}})+r_m^{\ast }({\mathbf{z}}_{\mathbf{i}\mathbf{,}\mathbf{m}})+1\}$ where $r_{common}({\mathbf{z}}_{\mathbf{i}\mathbf{,}\mathbf{m}})=z_{1i,m}^3+z_{2i,m}^3+z_{3i,m}^3+z_{4i,m}^3+z_{5i,m}^3$, ${\mathbf{\text{z}}}_{i,m}={(z_{1i,m},\dots ,z_{p_{m}i,m})}^T$, $r_1^{\ast }({\mathbf{z}}_{\mathbf{i}\mathbf{,}\mathbf{1}})=0$, $r_2^{\ast }({\mathbf{z}}_{\mathbf{i}\mathbf{,}\mathbf{2}})=z_{6i,2}^3+z_{7i,2}^3$, and $r_3^{\ast }({\mathbf{z}}_{\mathbf{i}\mathbf{,}\mathbf{3}})=z_{6i,3}^3$.

Case 1 was considered because the nonparametric function $r_m({\mathbf{z}}_{\mathbf{i}\mathbf{,}\mathbf{m}})$ has a complex form with nonlinear functions of $\mathbf{\text{z}}$'s and interactions of $\mathbf{\text{z}}$'s. Cases 2–3 were considered to investigate how our methods with a Gaussian kernel can capture misspecified functional forms, such as quadratic functions or cubic functions.

We compare the performance of BF‐GFKM with BF‐GAKM and FT‐SAKM for Cases 1–2 and with BF‐GAKM for Case 3.

4.1.2. Simulation Result

Table 1 summarized the ranges of $B{F}_{1,adj}$ obtained from adj‐BF method for three hypothesis tests ($H_{0,\ell }$ vs. $H_{1,\ell }$, $\ell =1,2,3$) of BF‐GFKM and BF‐GAKM in all cases. The simulation results for Case 1, Case 2, and Case 3 are summarized in Tables 2 and 3, Tables 4 and S1, and Tables S2 and 5, respectively. Note that the results for Case 2 with $(n,\mathbf{\text{p}})=(100,110,105,100)$ are presented in Table S1 of the Supporting Information, and the results for Case 3 with $(n,\mathbf{\text{p}})=(80,90,85,80)$ are presented in Table S2 of the Supporting Information. Since the results in Tables 4 and S1 are similar, as are those in Tables 5 and S2, we provide Tables S1 and S2 in the Supporting Information.

TABLE 1.

The ranges of $B{F}_{1,adj}$, which is the cutoff values of $BF$ for determining a significant set and an insignificant set from adj‐BF method for three hypothesis tests ($H_{0,m}$ vs. $H_{1,m}$, $m=1,2,3$) of BF‐GFKM and BF‐GAKM in all simulation Cases.

		(n,$p_1,p_2,p_3$)	Method	$H_{0,1}$ vs. $H_{1,1}$	$H_{0,2}$ vs. $H_{1,2}$	$H_{0,3}$ vs. $H_{1,3}$
				($U{B}_1$ $<B{F}_{1,adj}<$ $L{B}_2$)	($U{B}_1$ $<B{F}_{1,adj}<$ $L{B}_2$)	($U{B}_1$ $<B{F}_{1,adj}<$ $L{B}_2$)
Continuous	Case 1	(50,60,55,50)	BF‐GFKM	(10.326, 32.217)	(19,719, 26.923)	(5.634, 41.742)
			BF‐GAKM	(3.144, 16.672)	(9.326, 312.672)	(2.507, 6188.328)
		(100,110,105,100)	BF‐GFKM	(0.001, 120.373)	(0.093, 15.559)	(0.006, 1656.376)
			BF‐GAKM	(0.523, 5.581)	(1.355, 2.855)	($8.975\times 10^{-5}$, 4210.605)
	Case 2	(50,60,55,50)	BF‐GFKM	(8.173, 12.480)	(1.408, 611.794)	(9.748, 75.762)
			BF‐GAKM	(1.450, 2.166)	(0.056, 1133.734)	(0.679, 52.728)
		(100,110,105,100)	BF‐GFKM	($5.712\times 10^{-5}$, 6329.68)	(0.015, $5.719\times 10^9$)	(0.021, $2.363\times 10^{10}$)
			BF‐GAKM	(0.765, 45.074)	($1.755\times 10^{-4}$, 4379.843)	($9.597\times 10^{-4}$, 104.695)
Binary	Case 3	(80,90,85,80)	BF‐GFKM	(3.562, 3.909)	(2.059, 2.303)	(6.729, 7.390)
			BF‐GAKM	(6.211, 6.453)	(5.007, 5.276)	(7.480, 7.815)
		(100,110,105,100)	BF‐GFKM	(6.730, 7.200)	(1.489, 1.578)	(5.362, 5.704)
			BF‐GAKM	(7.800, 8.583)	(8.067, 9.490)	(6.777, 8.275)

TABLE 2.

The results of three hypothesis tests ($H_{0,m}$ vs. $H_{1,m}$, $m=1,2,3$) for Case 1 with $(n,\mathbf{\text{p}})=(n,p_1,p_2,p_3)=(50,60,55,50)$.

				Decision Rule
Setting	Hypothesis	Method	Criteria	$BF>1$	$BF>3$	$BF>5$	$BF>10$	$BF>B{F}_{1,adj}$
$n=50$, $p_1=60$, $p_2=55$, $p_3=50$	$H_{0,1}$ vs. $H_{1,1}$	BF‐GFKM	TPR	1	1	1	1	1
		BF‐GAKM		1	0.990	0.990	0.990	0.990
		FT‐SAKM		0.840
		BF‐GFKM	FPR	0.350	0.140	0.080	0.050	0.040
		BF‐GAKM		0.030	0.020	0.010	0.010	0.010
		FT‐SAKM		0.040
		BF‐GFKM	Accuracy	0.825	0.930	0.960	0.975	0.980
		BF‐GAKM		0.985	0.985	0.990	0.990	0.990
		FT‐SAKM		0.900
		BF‐GFKM	Precision	0.741	0.877	0.926	0.952	0.962
		BF‐GAKM		0.971	0.980	0.990	0.990	0.990
		FT‐SAKM		0.955
	$H_{0,2}$ vs. $H_{1,2}$	BF‐GFKM	TPR	1	1	1	1	1
		BF‐GAKM		1	1	1	1	1
		FT‐SAKM		0.840
		BF‐GFKM	FPR	0.360	0.170	0.150	0.100	0.060
		BF‐GAKM		0.070	0.020	0.010	0	0
		FT‐SAKM		0.190
		BF‐GFKM	Accuracy	0.820	0.915	0.925	0.950	0.970
		BF‐GAKM		0.965	0.990	0.995	1	1
		FT‐SAKM		0.825
		BF‐GFKM	Precision	0.735	0.855	0.870	0.909	0.943
		BF‐GAKM		0.935	0.980	0.990	1	1
		FT‐SAKM		0.816
	$H_{0,3}$ vs. $H_{1,3}$	BF‐GFKM	TPR	1	1	1	1	1
		BF‐GAKM		1	1	1	1	1
		FT‐SAKM		0.840
		BF‐GFKM	FPR	0.240	0.100	0.050	0.020	0.020
		BF‐GAKM		0.030	0	0	0	0
		FT‐SAKM		0.040
		BF‐GFKM	Accuracy	0.880	0.950	0.975	0.990	0.990
		BF‐GAKM		0.985	1	1	1	1
		FT‐SAKM		0.905
		BF‐GFKM	Precision	0.806	0.909	0.952	0.980	0.980
		BF‐GAKM		0.971	1	1	1	1
		FT‐SAKM		0.966

Note: The significance level $\alpha =0.05$ of FT‐SAKM. The ranges of $B{F}_{1,adj}$, which is the cutoff values of $BF$ to determine a significant set and an insignificant set for BF‐GFKM across three hypothesis tests, are (10.326, 32.217), (19.719, 26.923), and (5.634, 41.742), while those for BF‐GAKM are (3.144, 16.672), (9.326, 312.672), and (2.507, 6188.328).

TABLE 3.

The results of three hypothesis tests ($H_{0,m}$ vs. $H_{1,m}$, $m=1,2,3$) for Case 1 with $(n,\mathbf{\text{p}})=(n,p_1,p_2,p_3)=(100,110,105,100)$.

				Decision Rule
Setting	Hypothesis	Method	Criteria	$BF>1$	$BF>3$	$BF>5$	$BF>10$	$BF>B{F}_{1,adj}$
$n=100$, $p_1=110$, $p_2=105$, $p_3=100$	$H_{0,1}$ vs. $H_{1,1}$	BF‐GFKM	TPR	1	1	1	1	1
		BF‐GAKM		0.980	0.980	0.980	0.970	0.980
		FT‐SAKM		0.950
		BF‐GFKM	FPR	0	0	0	0	0
		BF‐GAKM		0	0	0	0	0
		FT‐SAKM		0.060
		BF‐GFKM	Accuracy	1	1	1	1	1
		BF‐GAKM		0.990	0.990	0.990	0.985	0.990
		FT‐SAKM		0.945
		BF‐GFKM	Precision	1	1	1	1	1
		BF‐GAKM		1	1	1	1	1
		FT‐SAKM		0.941
	$H_{0,2}$ vs. $H_{1,2}$	BF‐GFKM	TPR	1	1	1	1	1
		BF‐GAKM		0.960	0.940	0.920	0.890	0.950
		FT‐SAKM		0.950
		BF‐GFKM	FPR	0	0	0	0	0
		BF‐GAKM		0	0	0	0	0
		FT‐SAKM		0.100
		BF‐GFKM	Accuracy	1	1	1	1	1
		BF‐GAKM		0.980	0.970	0.960	0.945	0.975
		FT‐SAKM		0.950
		BF‐GFKM	Precision	1	1	1	1	1
		BF‐GAKM		1	1	1	1	1
		FT‐SAKM		0.909
	$H_{0,3}$ vs. $H_{1,3}$	BF‐GFKM	TPR	1	1	1	1	1
		BF‐GAKM		1	1	1	1	1
		FT‐SAKM		0.950
		BF‐GFKM	FPR	0	0	0	0	0
		BF‐GAKM		0	0	0	0	0
		FT‐SAKM		0.030
		BF‐GFKM	Accuracy	1	1	1	1	1
		BF‐GAKM		1	1	1	1	1
		FT‐SAKM		0.960
		BF‐GFKM	Precision	1	1	1	1	1
		BF‐GAKM		1	1	1	1	1
		FT‐SAKM		0.969

Note: The significance level $\alpha =0.05$ of FT‐SAKM. The ranges of $B{F}_{1,adj}$, which is the cutoff values of $BF$ to determine a significant set and an insignificant set for BF‐GFKM across three hypothesis tests, are (0.001, 120.373), (0.093, 15.559), and (0.006, 1656.376), while those for BF‐GAKM are (0.523, 5.581), (1.355, 2.855), and ($8.975\times 10^{-5}$, 4210.605).

TABLE 4.

The results of three hypothesis tests ($H_{0,m}$ vs. $H_{1,m}$, $m=1,2,3$) for Case 2 with $(n,\mathbf{\text{p}})=(n,p_1,p_2,p_3)=(50,60,55,50)$.

				Decision Rule
Setting	Hypothesis	Method	Criteria	$BF>1$	$BF>3$	$BF>5$	$BF>10$	$BF>B{F}_{1,adj}$
$n=50$, $p_1=60$, $p_2=55$, $p_3=50$	$H_{0,1}$ vs. $H_{1,1}$	BF‐GFKM	TPR	1	1	1	1	1
		BF‐GAKM		1	0.980	0.970	0.960	0.990
		FT‐SAKM		0.960
		BF‐GFKM	FPR	0.320	0.110	0.060	0.020	0.020
		BF‐GAKM		0.010	0	0	0	0
		FT‐SAKM		0.080
		BF‐GFKM	Accuracy	0.840	0.945	0.970	0.990	0.990
		BF‐GAKM		0.995	0.990	0.985	0.980	0.995
		FT‐SAKM		0.940
		BF‐GFKM	Precision	0.758	0.901	0.943	0.980	0.980
		BF‐GAKM		0.990	1	1	1	1
		FT‐SAKM		0.923
	$H_{0,2}$ vs. $H_{1,2}$	BF‐GFKM	TPR	1	1	1	1	1
		BF‐GAKM		1	1	1	1	1
		FT‐SAKM		0.960
		BF‐GFKM	FPR	0.010	0	0	0	0
		BF‐GAKM		0	0	0	0	0
		FT‐SAKM		0.140
		BF‐GFKM	Accuracy	0.995	1	1	1	1
		BF‐GAKM		1	1	1	1	1
		FT‐SAKM		0.910
		BF‐GFKM	Precision	0.990	1	1	1	1
		BF‐GAKM		1	1	1	1	1
		FT‐SAKM		0.873
	$H_{0,3}$ vs. $H_{1,3}$	BF‐GFKM	TPR	1	1	1	1	1
		BF‐GAKM		1	1	1	1	1
		FT‐SAKM		0.960
		BF‐GFKM	FPR	0.240	0.080	0.050	0.010	0.010
		BF‐GAKM		0	0	0	0	0
		FT‐SAKM		0.100
		BF‐GFKM	Accuracy	0.880	0.960	0.975	0.995	0.995
		BF‐GAKM		1	1	1	1	1
		FT‐SAKM		0.930
		BF‐GFKM	Precision	0.806	0.926	0.952	0.990	0.990
		BF‐GAKM		1	1	1	1	1
		FT‐SAKM		0.906

Note: The significance level $\alpha =0.05$ of FT‐SAKM. The ranges of $B{F}_{1,adj}$, which is the cutoff values of $BF$ to determine a significant set and an insignificant set for BF‐GFKM across three hypothesis tests, are (8.173, 12.480), (1.408, 611.794), and (9.748, 75.762), while those for BF‐GAKM are (1.450, 2.166), (0.056, 1133.734), and (9.748, 75.762).

TABLE 5.

The results of three hypothesis tests ($H_{0,m}$ vs. $H_{1,m}$, $m=1,2,3$) for Case 3 with $(n,\mathbf{\text{p}})=(n,p_1,p_2,p_3)=(100,110,105,100)$.

				Decision Rule
Setting	Method	Criteria	$BF>1$	$BF>1.5$	$BF>2$	$BF>B{F}_{1,adj}$
$n=100$, $p_1=110$, $p_2=105$, $p_3=100$	$H_{0,1}$ vs. $H_{1,1}$	BF‐GFKM	TPR	0.910	0.880	0.870	0.670
		BF‐GAKM		0.900	0.880	0.840	0.660
		BF‐GFKM	FPR	0.540	0.440	0.430	0.190
		BF‐GAKM		0.720	0.620	0.470	0.170
		BF‐GFKM	Accuracy	0.685	0.720	0.720	0.740
		BF‐GAKM		0.590	0.630	0.685	0.745
		BF‐GFKM	Precision	0.628	0.667	0.669	0.779
		BF‐GAKM		0.556	0.587	0.641	0.795
	$H_{0,2}$ vs. $H_{1,2}$	BF‐GFKM	TPR	0.750	0.700	0.670	0.700
		BF‐GAKM		0.940	0.930	0.880	0.630
		BF‐GFKM	FPR	0.320	0.250	0.200	0.250
		BF‐GAKM		0.660	0.590	0.530	0.280
		BF‐GFKM	Accuracy	0.715	0.725	0.735	0.725
		BF‐GAKM		0.640	0.670	0.675	0.675
		BF‐GFKM	Precision	0.701	0.737	0.770	0.737
		BF‐GAKM		0.588	0.612	0.624	0.692
	$H_{0,3}$ vs. $H_{1,3}$	BF‐GFKM	TPR	0.920	0.910	0.900	0.750
		BF‐GAKM		0.970	0.950	0.940	0.780
		BF‐GFKM	FPR	0.460	0.390	0.310	0.140
		BF‐GAKM		0.610	0.520	0.460	0.130
		BF‐GFKM	Accuracy	0.730	0.760	0.795	0.805
		BF‐GAKM		0.680	0.715	0.740	0.825
		BF‐GFKM	Precision	0.667	0.700	0.744	0.843
		BF‐GAKM		0.646	0.671	0.860	0.857

Note: The ranges of $B{F}_{1,adj}$, which is the cutoff values of $BF$ to determine a significant set and an insignificant set for BF‐GFKM across three hypothesis tests, are (6.730, 7.200), (1.489, 1.578), and (5.362, 5.704), while those for BF‐GAKM are (7.800, 8.583), (8.067, 9.490), and (6.777, 8.275).

The boxplots of four criteria for Cases 1–2 and Case 3 are displayed in Figure 3 and in Figure 4, respectively. In each Figure, we also display the four criteria of BF‐GFKM using the adj‐BF and traditional thresholds.

FIGURE 3.

The box plots of four criteria under simulation Cases 1–2. The orange, green, and blue box plots are obtained from BF‐GFKM and BF‐GAKM using various thresholds, including traditional BF thresholds and FT‐SAKM, respectively. The left top (a) is TPR; The right top (b) is FPR; The left bottom (c) is Accuracy. The right bottom (d) is Precision.

FIGURE 4.

The box plots of four criteria under simulation Case 3. The orange and sky blue box plots are obtained from BF‐GFKM and BF‐GAKM using various thresholds, including traditional BF thresholds. The left top (a) is TPR; The right top (b) is FPR; The left bottom (c) is Accuracy. The right bottom (d) is Precision.

Table 2 summarized the three hypothesis tests ($H_{0,\ell }$ vs. $H_{1,\ell }$, $\ell =1,2,3$) for Case 1 with $(n,\mathbf{\text{p}})=(n,p_1,p_2,p_3)=(50,60,55,50)$. Among BF‐GFKM using various thresholds, BF‐GFKM using adj‐BF performs the best in terms of all criteria. The TPR values of BF‐GFKM using adj‐BF are $100\%$ for three hypothesis tests, while that of FT‐SAKM is $84\%$. The FPR values of BF‐GFKM of three tests are $4\%,6\%$, and $2\%$, those of BF‐GAKM are $1\%,0\%$, and 0%, and those of FT‐SAKM are $4\%,19\%$, and 4%. The Accuracy values of BF‐GFKM are $98\%,97\%$, and $99\%$, and those of BF‐GAKM are $99\%,100\%$, and $100\%$, which are larger than those of FT‐SAKM, $90\%,82.5\%$, and $90.5\%$. The Precision values of BF‐GFKM are $96.2\%,94.3\%,98\%$, those values of BF‐GAKM are $99\%,100\%,100\%$, and those values of FT‐SAKM are $95.5\%,81.6\%,96.6\%$.

The ranges of $B{F}_{1,adj}$ for BF‐GFKM across three hypothesis tests are (10.326, 32.217), (19.719, 26.923), and (5.634, 41.742), while those for BF‐GAKM are (3.144, 16.672), (9.326, 312.672), and (2.507, 6188.328). The BF‐GFKM and BF‐GAKM perform better than FT‐SAKM in terms of TPR, Accuracy, and Precision. The two approaches are comparable in terms of FPR since the FPR of the two approaches is the same for the first hypothesis testing ($H_{0,1}$ vs. $H_{1,1}$).

On the other hand, Table 3 contains the multiple testing results for Case 1 with $(n,\mathbf{\text{p}})=(n,p_1,p_2,p_3)=(100,110,105,100)$. We can observe that TPR, Accuracy, and Precision are increased, and FPR is decreased for BF‐GFKM and FT‐SAKM, as we expected, because of the larger sample size. However, the TPR and Accuracy of BF‐GAKM were slightly reduced, but FPR and Precision are better than those of a small sample size. The performance of BF‐GFKM and BF‐GAKM is better than that of FT‐SAKM in terms of the four criteria. The ranges of $B{F}_{1,adj}$ for BF‐GFKM across three hypothesis tests are (0.001, 120.373), (0.093, 15.559), and (0.006, 1656.376), while those for BF‐GAKM are (0.523, 5.581), (1.355, 2.855), and ($8.975\times 10^{-5}$, 4210.605).

Table 4 displayed hypothesis testing results for Case 2 with $(n,\mathbf{\text{p}})=(n,p_1,p_2,p_3)=(50,60,55,50)$, whereas Table S1 summarized the results for Case 2 with $(n,\mathbf{\text{p}})=(n,p_1,p_2,p_3)=(100,110,105,100)$. The results for Case 2 are similar to those for Case 1. BF‐GFKM and BF‐GAKM perform better than the FT‐SAKM in terms of the four criteria. The ranges of $B{F}_{1,adj}$ for BF‐GFKM across three hypothesis tests with small sample sizes are (8.173, 12.480), (1.408, 611.794), and (9.748, 75.762), while those for BF‐GAKM are (1.450, 2.166), (0.056, 1133.734), and (9.748, 75.762). The ranges of $B{F}_{1,adj}$ for BF‐GFKM across three hypothesis tests with large sample size are ($5.712\times 10^{-5}$, 6329.68), (0.015, $5.719\times 10^9$), and (0.021, $2.363\times 10^{10}$), while those for BF‐GAKM are ($5.712\times 10^{-5}$, 6329.68), (0.015, $5.719\times 10^9$), and (0.021, $2.363\times 10^{10}$). Among BF‐GFKM using various thresholds, BF‐GFKM using adj‐BF performs the best in terms of all criteria. Also, we can observe that BF‐GAKM using adj‐BF performs better than that using other traditional thresholds. The simulation results suggest that the performances of the three approaches for Case 2 are better than those for Case 1 because the function of Case 2 has only a quadratic form of z's, whereas the function of Case 1 has a more complex form of z's. The box plots of four criteria of BF‐GFKM using adj‐BF and traditional thresholds and other comparison methods for simulation Cases 1–2 are outlined in Figure 3. Overall, the performance of BF‐GFKM and BF‐GAKM using adj‐BF is better than that of FT‐SAKM.

The results on binary response variable for Case 3 with $(n,\mathbf{\text{p}})=(n,p_1,p_2,p_3)=(80,90,85,80)$ and with $(n,\mathbf{\text{p}})=(n,p_1,p_2,p_3)=(100,90,85,80)$ are summarized in Tables S2 and 5. Among BF‐GFKM using various thresholds, BF‐GFKM using adj‐BF performs the best in terms of all criteria. Also, we can observe that BF‐GAKM using adj‐BF performs better than that using other traditional thresholds. The TPR values of BF‐GFKM using adj‐BF are $80\%,68\%$, and $74\%$ for three hypothesis tests, whereas those of BF‐GAKM are $74\%,67\%$, and $67\%$. The FPR values of BF‐GFKM of three tests are $18\%,29\%,16\%$, and those of BF‐GAKM are $27\%,42\%,19\%$. The Accuracy values of BF‐GFKM are $81\%,69.5\%$, and $79\%$, which are larger than those of BF‐GAKM: $73.5\%,62.5\%$, and $74\%$. The Precision values of BF‐GFKM are $81.6\%,70.1\%$, and $82.2\%$, and those values of BF‐GAKM are $73.3\%,61.5\%$, and $77.9\%$. The ranges of $B{F}_{1,adj}$ for BF‐GFKM across three hypothesis tests are (3.562, 3.909), (2.059, 2.303), and (6.729, 7.390), while those for BF‐GAKM are (6.211, 6.453), (5.007, 5.276), and (7.480, 7.815). The results on binary outcomes for larger sample sizes are similar to those for smaller ones. However, the performance of BF‐GFKM is worse than that of BF‐GAKM for $H_{0,1}$ versus $H_{1,1}$ in terms of FPR, Accuracy, and Precision. Also, BF‐GFKM performs worse than BF‐GAKM for the third hypothesis testing regarding the four criteria. The overall performance of BF‐GFKM on binary outcomes for a larger sample size is comparable to that of BF‐GAKM. The ranges of $B{F}_{1,adj}$ for BF‐GFKM across three hypothesis tests are (6.730, 7.200), (1.489, 1.578), and (5.362, 5.704), while those for BF‐GAKM are (7.800, 8.583), (8.067, 9.490), and (6.777, 8.275). The box plots of four criteria of BF‐GFKM using adj‐BF and traditional thresholds and other comparison methods for simulation Case 3 are also outlined in Figure 4. Again, we observe that BF‐GFKM performs better than BF‐GAKM in terms of the four criteria for Case 3. BF‐GFKM using adj BF performs better than that using other thresholds.

4.2. Independent Sets Without Shared Elements

This section presents a simulation study designed to investigate the performance of our approach under the condition of no correlation among sets. The core setup remains the same as in Section 4.1.1.

4.2.1. Setting

We consider Case 1 for testing three hypotheses ($H_{0,m}$ vs. $H_{1,m}$, $m=1,2,3$) with the setting $(n,\mathbf{\text{p}})=(n,p_1,p_2,p_3)=(50,60,55,50)$. For each case, we ran 100 simulations and conducted 10 000 MCMC iterations.

We first generated the covariates ${\mathbf{\text{x}}}_i={(x_{i1},x_{i2})}^T,x_{i1}\sim N(10,1),x_{i2}\sim Ber(0.5)$, and set ${\mathit{\beta }}_{true}=(1,1)$. We considered $r_m({\mathbf{z}}_{\mathbf{i}\mathbf{,}\mathbf{m}})=0.5\{r_{common}({\mathbf{z}}_{\mathbf{i}\mathbf{,}\mathbf{m}})+r_m^{\ast }({\mathbf{z}}_{\mathbf{i}\mathbf{,}\mathbf{m}})\}$ for $m=1,2,3$, which contains two parts: shared common ($r_{commom}$) and nonshared ($r_m^{\ast }$) parts. We also generated $\mathbf{\text{Z}}=({\mathbf{\text{Z}}}_1,\dots ,{\mathbf{\text{Z}}}_M)$ from $N(\mathbf{\text{0}},\sum )$ with the correlation matrix $\sum =A⨂I_M$, where $A=corr({\mathbf{\text{z}}}_{i,m},{\mathbf{\text{z}}}_{j,m})=0.2{\mathbf{\text{I}}}_n+0.8J_M$, $J_M$ is an $M\times M$ matrix with all‐ones components, and $I_M$ is the $M\times M$ identity matrix. This means there is no correlation among sets. Each $z_{ki,m}$ was generated from N(0,1) and then transformed to Uniform(−2,2). To generate a continuous response variable, we used the function of Case 1 in Section 4.1.1.

4.2.2. Simulation Result

Table S3 of Supporting Information presents the results of three hypothesis tests for Case 1 in a setting where there is no correlation between sets. Both BF‐GFKM and BF‐GAKM achieved a near‐perfect performance with the adj‐BF decision rule. The BF‐GFKM attained a TPR of 1, FPR of 0, Accuracy of 1, and Precision of 1 across all three hypotheses. The BF‐GAKM also achieved similar results to those of BF‐GFKM except for one hypothesis ($H_{0,2}$ vs. $H_{1,2}$). The FT‐SAKM also performs much better in this setting. Its TPR improve to $99\%$ (up from $84\%$ in the correlated setting), while its FPR drops to $7\%$ for the first test ($H_{0,1}$ vs. $H_{1,1}$) and $5\%$ for the other two tests. Consequently, its Accuracy and precision values also see a notable increase, reaching $96-97\%$ and $93.4-95.2\%$, respectively.

The improved performance of the FT‐SAKM suggests that the presence of correlation between sets in the data generation process significantly hinders the frequentist approach's ability to accurately test the hypotheses.

4.3. Correlated Sets With Shared Elements

This section details a simulation study designed to investigate the performance of our approach under a shared element setting. The core setup remains the same as in Section 4.1.1.

4.3.1. Setting

Specifically, we consider Case 1 for testing three hypotheses ($H_{0,m}$ vs. $H_{1,m},m=1,2,3$), with the setting $(n,\mathbf{p})=(n,p_1,p_2,p_3)=(50,60,55,50)$. The covariates ${\mathbf{\text{x}}}_i={(x_{i1},x_{i2})}^T$ were generated from $x_{i1}\sim N(10,1),x_{i2}\sim Ber(0.5)$, with ${\mathit{\beta }}_{true}=(1,1)$.

The correlation structure is the same as in Section 4.1.1. The significant change in this setting is the introduction of a common element between the first two sets of variables, achieved by modifying the functional form for Case 1 as follows:

• •
  Case 1: $y_i={\mathbf{\text{x}}}_i^T\mathit{\beta }+{\sum }_{m=1}^3{r}_m({\mathbf{\text{z}}}_{i,m})+{ϵ}_i$ with ${ϵ}_i\sim N(0,1)$, $r_m({\mathbf{z}}_{\mathbf{i}\mathbf{,}\mathbf{m}})=0.5\{r_{common}({\mathbf{z}}_{\mathbf{i}\mathbf{,}\mathbf{m}})+r_m^{\ast }({\mathbf{z}}_{\mathbf{i}\mathbf{,}\mathbf{m}})\}$ where $r_{common}({\mathbf{z}}_{\mathbf{i}\mathbf{,}\mathbf{m}})=z_{1i,m}^3-z_{2i,m}^2+z_{1i,m}+z_{2i,m}$, and ${\mathbf{\text{z}}}_{i,m}={(z_{1i,m},\dots ,z_{p_{m}i,m})}^T$.

  • ·
    $r_1^{\ast }({\mathbf{z}}_{\mathbf{i}\mathbf{,}\mathbf{1}})=-0.5exp(z_{3i,1})cos(z_{4i,1})+z_{5i,1}-z_{6i,1}+sin(z_{7i,1})+cos(z_{8i,m})+0.5z_{9i,1}{z}_{10i,1}$
  • ·
    $r_2^{\ast }({\mathbf{z}}_{\mathbf{i}\mathbf{,}\mathbf{2}})=-0.5exp(z_{3i,2})cos({\mathbf{z}}_{\mathbf{4}\mathbf{i}\mathbf{,}\mathbf{1}})+sin(z_{5i,2})-z_{6i,2}+sin(z_{7i,2})+cos{(z_{8i,2})}^2+0.5z_{9i,2}{z}_{10i,2}+cos{(z_{11i,2})}^2+0.5z_{12i,2}{z}_{13i,2}+cos{(z_{14i,2})}^2$
  • ·
    $r_3^{\ast }({\mathbf{z}}_{\mathbf{i}\mathbf{,}\mathbf{3}})=-0.5exp(z_{3i,3})z_{4i,3}+sin{(z_{6i,3})}^2+0.5sin(z_{4i,3})cos(z_{5i,3})+z_{7i,3}^2+z_{7i,3}$

The key modification is in the expression for $r_2^{\ast }$, where the term $cos(z_{4i,2})$ has been replaced with $cos({\mathbf{z}}_{\mathbf{4}\mathbf{i}\mathbf{,}\mathbf{1}})$. This change introduces a single common element from the first set of variables ($z_{4i,1}$) into the functional form of the second set ($r_2^{\ast }({\mathbf{z}}_{\mathbf{i}\mathbf{,}\mathbf{2}})$). This simulation is designed to evaluate how each method performs when a true underlying relationship between sets exists through a shared element.

4.3.2. Simulation Result

Table S4 of Supporting Information presents the simulation results for Case 1 under a common element setting. As in previous simulations, we evaluate the performance of BF‐GFKM, BF‐GAKM, and FT‐SAKM methods based on their TPR, FPR, Accuracy, and Precision. The core distinction in this simulation is the introduction of a shared element between the first and second sets.

The results show that both Bayesian methods, BF‐GFKM and BF‐GAKM, maintain exceptional performance. For all three hypotheses, BF‐GAKM achieved a TPR of 1 and FPR of 0, resulting in a perfect Accuracy and Precision of 1. BF‐GFKM also achieved the same performance except for the second hypothesis test ($H_{0,2}$ vs. $H_{1,2}$), but the performance of BF‐GFKM in the second hypothesis test also is close to TPR of 1 and FPR of 0.

The frequentist FT‐SAKM method also demonstrates strong performance, with a consistent TPR of 0.850 across all three hypotheses. While this is a good result, it is still lower than the near‐perfect TPR achieved by the Bayesian methods. The FPR for FT‐SAKM is notably lower in the third hypothesis ($H_{0,3}$ vs. $H_{1,3}$) at 0.020, but higher for the first and second hypotheses (0.120 and 0.180, respectively). This variability suggests that the performance of the frequentist method is more sensitive to the specific functional form of the relationship, even in the presence of a common element.

5. Type II Diabetes Genetic‐Pathways Data

We applied our BF‐GFKM to type II diabetes genetic‐pathways data [1]. These data include 18 type II diabetes patients and 17 normal glucose levels. Several studies [8, 28] also reported significant multiple pathways either using a score test [8] or a hybrid test [28]. Their analysis is a single‐pathway analysis, ignoring dependence among pathways. Pang et al. [8] identified significant pathways associated with the glucose level, whereas Xu et al. [28] detected important pathways to distinguish between normal glucose‐level individuals and those with type II diabetes. In our study, we collected 21 pathways from these two studies.

A systematic selection framework was adopted to delineate pathways recurrently identified across both primary sources. Specifically, (i) all pathways reported as significant in both reference studies were retained; (ii) two pathways consistently classified as nonsignificant in both studies were incorporated to preserve representational balance; and (iii) six to eleven uniquely significant pathways identified in each study were additionally included to ensure comprehensive coverage. This procedure yielded a target set of 18–24 high‐priority pathways (corresponding to approximately 6–8 composite combinations). Selection of fewer combinations would curtail representation of validated candidates, whereas expansion beyond this range would necessitate inclusion of marginal pathways and introduce excessive computational and inferential burden, particularly under the limited sample size ($n=35$). Guided by established metabolic and signaling interdependencies, the final configuration comprised seven composite combinations, partitioning the 21 pathways into seven functionally coherent clusters.

In our analysis, we considered two types of response variables: one for a continuous outcome, the log‐transformed glucose level, and the other for a binary outcome, representing the type II diabetes status. We fit BF‐GFKM using three pathways ($M=3$). Let Z be the $\mathbf{\text{p}}\times n$ gene expression, where $n$ is 35, and $\mathbf{\text{p}}=(p_1,p_2,p_3)$, which varies from 16 to 140. Our goal is to identify significantly correlated pathways that affect the glucose level and pathways to distinguish between normal and type II diabetes patients. We decided on cutoff values of $BF$ using the adj‐BF method for multiple testing.

Our analysis consists of two parts. The first is to conduct seven combinations of 21 pathways, summarized in Table 6. For each combination, we conducted the hypothesis tests. The second part is to conduct all possible three combinations out of the three pathways. Although the total number of orders for these three pathways is $3!$, the total number of tests is 3 ($3!/2$) because some of the model estimations of the ordering of the three pathways are the same, as we explain in Section 3.2. We consider two combinations of three pathways shown in Table 8.

TABLE 6.

The seven combinations out of 21 pathways.

Combination	Pathway	Name
1	1	Alanine and aspartate metabolism
	2	ATP synthesis
	3	c22 U133 probes(user defined)
2	1	c23 U133 probes(user defined)
	2	JAK‐Stat Signaling Pathway
	3	Oxidative phosphorylation
3	1	OXPHOS HG‐U133A probes
	2	Parkinson's disease
	3	Ubiquinone biosynthesis
4	1	gamma‐Hexachlorocyclohexane degradation
	2	MAP00561 Glycerolipid metabolism
	3	Histidine metabolism
5	1	Fructose and mannose metabolism
	2	MAP00071 Fatty acid metabolism
	3	MAP00190 Oxidative phosphorylation
6	1	MAP00380 Tryptophan metabolism
	2	Starch and sucrose metabolism
	3	Wnt signaling pathway
7	1	Ubiquitin mediated proteolysis
	2	Apoptosis
	3	Complement and coagulation cascades

TABLE 8.

The three orderings for two combinations of three pathways chosen from 5 pathways.

Combination	Ordering	Pathway	Name
1	1	1	ATP synthesis
		2	Ubiquitin mediated proteolysis
		3	JAK‐Stat Signaling Pathway
	2	1	Ubiquitin mediated proteolysis
		2	ATP synthesis
		3	JAK‐Stat Signaling Pathway
	3	1	ATP synthesis
		2	JAK‐Stat Signaling Pathway
		3	Ubiquitin mediated proteolysis
2	1	1	Oxidative phosphorylation
		2	gamma‐Hexachlorocyclohexane degradation
		3	JAK‐Stat Signaling Pathway
	2	1	Oxidative phosphorylation
		2	JAK‐Stat Signaling Pathway
		3	gamma‐Hexachlorocyclohexane degradation
	3	1	JAK‐Stat Signaling Pathway
		2	Oxidative phosphorylation
		3	gamma‐Hexachlorocyclohexane degradation

5.1. Identifying Significant Pathways Associated With Glucose Level

Using our adj‐BF method, we first decided on $B{F}_{g,adj}$ values, $g=1,2,3$. We found that there were $G^{\ast }=4$ classes to decide $B{F}_{g,adj}$ of $BF\mathrm{s}$. The ranges of $B{F}_{1,adj}$, $B{F}_{2,adj}$, and $B{F}_{3,adj}$ of $BF\mathrm{s}$ obtained from the randomly chosen three combinations out of 21 pathways are (3.638, 4.278), (7.092, 10.452), and (17.948, 23.315), respectively. The total number of combinations was 9 (7 combinations have no order, but the other two combinations have three orderings), as shown in Tables 6, 7, 8.

TABLE 7.

$BF$ values of pathways in each combination: We calculated BF 100 times from 100 different MCMC samples.

Id	Name of pathway	#genes	Combination	Continuous (BF)	Binary (BF)
4	Alanine and aspartate metabolism	18	1	$✓$(5.733)	${✓}^{\ast \ast }$ (10.797)
16	ATP synthesis	49	1	${✓}^{\ast \ast }$(60.730)	1.391
43	c22 U133 probes(user defined)	95	1	1.789	$✓$(4.658)
44	c23 U133 probes(user defined)	67	2	1.566	${✓}^{\ast }$ (6.718)
110	JAK‐Stat Signaling Pathway	71	2	$✓$(5.326)	$✓$(4.727)
229	Oxidative phosphorylation	133	2	${✓}^{\ast \ast }$ (77.441)	$✓$(3.726)
230	OXPHOS HG‐U133A probes	121	3	${✓}^{\ast \ast }$	${✓}^{\ast }$ (6.956)
232	Parkinson's disease	40	3	3.121	${✓}^{\ast \ast }$ (9.518)
272	Ubiquinone biosynthesis	16	3	0.426	0.473
88	gamma‐Hexachlorocyclohexane degradation	41	4	$✓$(7.092)	$✓$(3.774)
177	MAP00561 Glycerolipid metabolism	43	4	${✓}^{\ast \ast }$ (23.316)	${✓}^{\ast }$ (7.752)
103	Histidine metabolism	37	4	${✓}^{\ast \ast }$ (39.060)	$✓$(3.585)
121	Fructose and mannose metabolism	23	5	3.638	2.781
126	MAP00071 Fatty acid metabolism	65	5	2.157	${✓}^{\ast }$ (8.269)
133	MAP00190 Oxidative phosphorylation	58	5	${✓}^{\ast }$ (17.948)	$✓$(4.976)
154	MAP00380 Tryptophan metabolism	60	6	${✓}^{\ast }$ (11.537)	${✓}^{\ast }$(7.449)
259	Starch and sucrose metabolism	54	6	$✓$(5.351)	2.840
278	Wnt signaling pathway	140	6	$✓$(10.452)	${✓}^{\ast }$ (6.059)
273	Ubiquitin mediated proteolysis	46	7	$✓$(6.070)	2.643
13	Apoptosis	92	7	${✓}^{\ast \ast }$ (29.370)	${✓}^{\ast }$ (7.904)
71	Complement and coagulation cascades	47	7	2.542	$✓$(3.635)

Note: The ranges of $B{F}_{1,adj}$, $B{F}_{2,adj}$, and $B{F}_{3,adj}$ of $BF\mathrm{s}$ for continuous are (3.638, 4.278), (7.092, 10.452), and (17.948, 23.315), respectively. The ranges of those of BF for binary are (2.890, 3.097), (4.977, 6.058), and (9.004, 9.517), respectively. $✓$, ${✓}^{\ast }$, and ${✓}^{\ast \ast }$ mean the pathway is significant, strongly significant, or very strongly significant, respectively.

First, for each combination, we conducted BF‐GFKM to test the hypothesis ($H_{0,\ell }$ vs. $H_{1,\ell }$, $\ell =1,2,3$). The testing result is summarized in Tables 7, 8, 9. In total, the 16 pathways are insignificant, whereas 23 pathways are significant, strongly significant, or very strongly significant. Table 7 shows $BF\mathrm{s}$ of 21 pathways with combinations 1–7. The largest $BF$ was 77.443, in which the BF of “Oxidative phosphorylation,” namely pathway 229, and combination 2. Mootha et al. [1] found that the “Oxidative phosphorylation” pathway is coordinately decreased in human diabetic muscle. It has been hypothesized that PGC‐1α, which is expressed in skeletal muscle and powerfully induces mitochondrial biogenesis when expressed ectopically in skeletal and cardiac myocytes, activates the “Oxidative phosphorylation” pathway [29]. “ATP synthesis” and “OXPHOS HG‐U133A probes” are very strongly significant pathways because the “ATP synthesis” subset of the “Oxidative phosphorylation” and “OXPHOS HG‐U133A probes” are superset of “Oxidative phosphorylation” [1]. The “JAK‐Stat Signaling Pathway” is associated with glucose levels as its inhibition has been shown to prevent glucose‐induced growth in glomerular mesangial cells [30]. The “Ubiquitin mediated proteolysis” is associated with glucose levels without the inclusion of the “JAK‐Stat Signaling Pathway” and “ATP synthesis” in three pathways models [8]. The pathways 13, 88, 103, 133, 154, 177, 259, and 278 are identified and show that these pathways are associated with glucose levels. These pathways are identified to distinguish between normal and type II diabetes patients [28].

TABLE 9.

$BF$ values of pathways in each combination: We calculated BF 100 times from 100 different MCMC samples.

ID	NAME	#genes	Combination	Ordering	Continuous	Binary
16	ATP synthesis	49	1	1	${✓}^{\ast \ast }$(44.197)	$✓$(3.935)
				2	${✓}^{\ast \ast }$(49.142)	$✓$(3.098)
				3	${✓}^{\ast \ast }$(43.051)	2.587
273	Ubiquitin mediated proteolysis	46		1	3.338	1.463
				2	2.723	2.890
				3	2.996	1.977
110	JAK‐Stat Signaling Pathway	71		1	3.565	${✓}^{\ast \ast }$(12.239)
				2	3.237	${✓}^{\ast \ast }$(11.456)
				3	$✓$(4.789)	${✓}^{\ast }$(7.251)
			2	1	$✓$(4.278)	${✓}^{\ast \ast }$(11.791)
				2	$✓$(4.453)	${✓}^{\ast }$(6.890)
				3	3.557	${✓}^{\ast }$(9.004)
88	gamma‐Hexachlorocyclohexane degradation	41		1	1.967	1.776
				2	1.706	$✓$(3.436)
				3	1.478	$✓$(3.332)
229	Oxidative phosphorylation	133		1	${✓}^{\ast \ast }$(172.314)	$✓$(4.918)
				2	${✓}^{\ast \ast }$(163.079)	$✓$(4.741)
				3	${✓}^{\ast \ast }$(134.874)	$✓$(3.229)

Note: The ranges of $B{F}_{1,adj}$, $B{F}_{2,adj}$, and $B{F}_{3,adj}$ of $BF\mathrm{s}$ for continuous are (3.638, 4.278), (7.092, 10.452), and (17.948, 23.315), respectively. The ranges of those of BF for binary are (2.890, 3.097), (4.977, 6.058), and (9.004, 9.517), respectively. $✓$, ${✓}^{\ast }$, and ${✓}^{\ast \ast }$ mean the pathway is significant, strongly significant, or very strongly significant, respectively.

Second, on the other hand, we investigated the ordering of three pathways that affect glucose levels. We selected five pathways and then considered three orderings from two combinations of five pathways. The results are summarized in Table 9. The “Ubiquitin mediated proteolysis” pathway is not a significant pathway with the inclusion of “ATP synthesis” and “JAK‐Stat Signaling Pathway”. The “Ubiquitin mediated proteolysis” pathway was not significant after the inclusion of “ATP synthesis” and “JAK‐Stat Signaling Pathway” under the additive model [8]. It appears that the pathways “Ubiquitin‐mediated proteolysis” and “JAK‐Stat Signaling Pathway” have significant crosstalk potential or are functionally related concerning glucose levels. These two pathways do not have any overlapping genes. However, the origin of this pathway from KEGG shows that “JAK‐Stat Signaling Pathway” pathway has linkage with “Ubiquitin‐mediated proteolysis” [8, 31].

“JAK‐Stat Signaling Pathway” with orderings 1–2 of combination 1 and ordering 3 of combination 2 is not a significant pathway. However, this pathway with ordering 3 of combination 1 and orderings 1–2 of combination 2 is significant. We can see that the testing results for glucose levels can be changed depending on the ordering of pathways.

The testing results of combinations 1, 2, and 4 and ordering 2 from combination 1 of the three pathways associated with the glucose level are in Figure 5. Figure 6 shows the testing results of orderings 1 and 3 for combinations 1–2 of three pathways associated with the glucose level. The solid blue circle indicates a significant pathway, whereas the dashed white circle indicates an insignificant pathway. The solid line denotes two pathways that are correlated with the fused structure, whereas the dashed line denotes that they are not.

FIGURE 5.

The hypothesis testing results of type II diabetes genetic pathway data. The solid blue circle indicates a significant pathway, while the dashed white circle indicates an insignificant pathway. The solid line denotes that two pathways are correlated with a fused structure, while the dashed line denotes that they are not. The left top (a) represents the result of combination 1 of three pathways associated with glucose level: pathways 4 and 16 correlate with fused dependence structure, but pathways 16 and 43 do not; pathways 4 and 16 are significant pathways to the glucose level. The right top (b) represents the result of combination 2 of three pathways associated with glucose level: pathways 110 and 229 are correlated with fused dependence structure, but pathways 110 and 44 are not; pathways 110 and 229 are significant pathways to glucose level related to diabetes. The bottom left (c) represents the result of combination 4 of three pathways associated with glucose level: pathways 88 and 177, as well as pathways 110 and 103, are correlated with a fused dependence structure. The right bottom (d) represents the result of ordering 2 for combination 1 of three pathways associated with glucose level: pathways 110 and 16, as well as pathways 110 and 273, are not correlated; pathway 110 is a significant pathway to the glucose level.

FIGURE 6.

The hypothesis testing results of type II diabetes genetic pathway data. The solid blue circle indicates a significant pathway, while the dashed white circle indicates an insignificant pathway. The solid line denotes that two pathways are correlated with a fused structure, while the dashed line denotes that they are not. The left top (a) represents the result of ordering 1 for combination 1 of three pathways associated with glucose level: pathways 16 and 273, as well as pathways 273 and 110, are not correlated; pathway 16 is a significant pathway to the glucose level. The right top (b) represents the result of ordering 3 for combination 1 of three pathways associated with glucose level: neither pathways 16 and 110 nor pathways 110 and 273 have correlation; pathways 16 and 273 are significant pathways to glucose level related to diabetes. The bottom left (c) represents the result of ordering 1 for combination 2 of three pathways associated with glucose level: pathways 229 and 88, as well as pathways 88 and 110, are not correlated; pathways 229 and 110 are significant. The right bottom (d) represents the result of ordering 3 for combination 2 of three pathways associated with glucose level: pathways 110 and 229, along with pathways 229 and 88, show no correlation; pathway 229 is a significant pathway to the glucose level.

5.2. Identifying Significant Pathways to Distinguish Normal and Type II Diabetes Patients

Again, we considered 21 pathways to identify significant pathways distinguishing between normal and type II diabetes patients. Using the adj‐BF method for multiple testing. We estimated $G^{\ast }=4$ classes to decide $B{F}_{g,adj}$ of $BF\mathrm{s}$, $g=1,2,3$,finding that the ranges of $B{F}_{1,adj}$, $B{F}_{2,adj}$, and $B{F}_{3,adj}$ of $BF\mathrm{s}$ are (2.890, 3.097), (4.977, 6.058), and (9.004, 9.517), respectively.

The testing results are in Tables 7, 8, 9. In total, the 14 pathways are insignificant, whereas 25 pathways are significant, strongly significant, or very strongly significant.

First, for each combination, we conducted BF‐GFKM to test the hypothesis ($H_{0,\ell }$ vs. $H_{1,\ell }$, $\ell =1,2,3$). Table 7 shows the $BF\mathrm{s}$ of 21 pathways with combinations 1–7. “Alanine and aspartate metabolism” has the largest $BF$ value 10.797. “Alanine and aspartate metabolism” was a significant pathway in Xu et al. [28]. “Oxidative phosphorylation” and “OXPHOS HG‐U133A probes” were significant pathways, but “ATP synthesis” with combination 1 was not significant. “Parkinson's disease” and “MAP00071 Fatty acid metabolism” were not significant for glucose levels, but were significant to distinguish between normal and diabetes patients [28].

Second, we also investigated the ordering of three pathways that affect binary outcomes. Using five pathways, we conducted our BF‐GFKM to test the hypothesis ($H_{0,\ell }$ vs. $H_{1,\ell }$, $\ell =1,2,3$). These testing results are summarized in Table 9. The “Ubiquitin mediated proteolysis” pathway is not a significant pathway for the binary phenotype, as well as for continuous glucose levels, with the inclusion of “ATP synthesis” and “JAK‐Stat Signaling Pathway.” The “JAK‐Stat Signaling Pathway” pathway is significant with orderings 1–3 for combinations 1–2. “ATP synthesis” is not significant with ordering 3 of combination 1, whereas it is significant with orderings 1–2 of combination 1. The “gamma‐Hexachlorocyclohexane degradation” pathway is significant with orderings 2–3 of combination 2, whereas it is not significant with ordering 3 of combination 2.

5.3. Significant Pathways for Continuous and Binary Responses

The significant pathways for both continuous and binary responses are in Tables 10 and 11. Pathways 4, 13, 88, 103, 110, 133, 154, 177, 229, 230, and 278 are significant pathways associated with glucose levels and distinguish between normal and type II diabetes patients among 21 pathways with seven combinations. Pathway 16 with orderings 1–2 of combination 1, pathway 110 with ordering 3 of combination 1, and orderings 1–2 of combination 2, and pathway 229 with orderings 1–3 of combination 2 are significant pathways associated with the glucose level and distinguish between normal and type II diabetes patients among five pathways with three orderings of two combinations.

TABLE 10.

The significant pathways, both continuous response and binary response, among 21 pathways with 7 combinations.

ID	NAME	#genes	Combination
4	Alanine and aspartate metabolism	18	1
110	JAK‐Stat Signaling Pathway	71	2
229	Oxidative phosphorylation	133	2
230	OXPHOS HG‐U133A probes	121	3
88	gamma‐Hexachlorocyclohexane degradation	41	4
177	MAP00561 Glycerolipid metabolism	43	4
103	Histidine metabolism	37	4
133	MAP00190 Oxidative phosphorylation	58	5
154	MAP00380 Tryptophan metabolism	60	6
278	Wnt signaling pathway	140	6
13	Apoptosis	92	7

TABLE 11.

The significant pathways for both continuous response and binary response among 5 pathways with 3 orderings of 2 combinations.

ID	NAME	#genes	Combination	Ordering
16	ATP synthesis	49	1	1
				2
110	JAK‐Stat Signaling Pathway	71		3
			2	1
				2
229	Oxidative phosphorylation	133		1
				2
				3

The test results of the “JAK‐Stat Signaling Pathway” for binary response are not changed by orderings 1–3 for combinations 1–2, but the test results of “ATP synthesis” and “gamma‐Hexachlorocyclohexane degradation” pathway for binary response are changed depending on orderings. We can consider that the “JAK‐Stat Signaling Pathway” is not affected by the other pathways, but the “ATP synthesis” and “gamma‐Hexachlorocyclohexane degradation” pathways are affected by other pathways. Therefore, if researchers are interested in determining whether specific pathways, such as the “ATP synthesis” and “gamma‐Hexachlorocyclohexane degradation” pathway, are influenced by other pathways, unlike “JAK‐Stat Signaling Pathway”, our BF‐GFKM method can be utilized to provide valuable insights into these relationships.

6. Conclusion

In this article, we have developed a flexible Bayesian inference based on BF using generalized fused kernel machine regression to test significantly correlated high‐dimensional functions (pathways) with the response variable, which can be continuous or binary. We developed a data‐driven, flexible Bayesian inference for adjusting BF for multiple tests.

We apply our method to pathway‐based analysis. Because pathways depend on each other, it is essential to analyze multiple pathways together by incorporating correlated structures. We demonstrate the advantages of genetic pathway‐based analysis of type II diabetes. Although some pathways are identified as significant to glucose levels or diabetes, they need to be further validated biologically.

We further note that the fused‐lasso structure is not intended to encode biological ordering or mechanistic pathway relationships. Instead, it is employed as a parsimonious statistical coupling prior to stabilize inference across pathway‐specific latent models. While network‐based models have been extensively studied in high‐dimensional settings, including pathway analysis, the existing literature predominantly focuses on network estimation rather than formal hypothesis testing. Developing principled testing procedures for multilevel graph‐structured network models for pathway genetic analysis remains an important direction for future research. Our proposed Bayesian framework contributes to this direction by enabling hypothesis testing within a simple yet flexible dependence structure.

Since the dependency among pathways is unknown in our application, we designed the BF‐GFKM to incorporate correlation structures among three pathways at a time using the fused penalty, conducting all possible distinct tests—a manageable task when limited to three pathways simultaneously. Our approach balances computational feasibility with the ability of the fused structure to capture correlations among pathways. We also note that our GFKM can be fitted with a group penalty and an $AR(1)$ fused penalty under the assumption that only adjacent pathways are correlated. In this setting, the $m\text{th}$ pathway is correlated with the $(m+1)\text{th}$ pathway but not with the $(m+2)\text{th}$. By jointly modeling multiple pathways and accounting for this correlation structure, our method provides a more accurate detection of pathways significantly associated with the response. In contrast, traditional pathway‐based analyses typically adopt a myopic strategy that tests one pathway at a time and assumes independence across pathways, which can lead to inflated false positives and false negatives. However, as more pathways are incorporated, the dimension of ${\sum }_r$ increases, often causing numerical instability when computing its inverse.

In general, when $M$ pathways are considered and an ordering can be established from correlations among genes within pathways, the BF‐GFKM test needs to be performed only once. In the absence of such information, however, all distinct orderings must be examined. Because each ordering is equivalent to its reverse, the number of distinct tests is $M!/2$, which grows factorially with $M$ and quickly becomes computationally infeasible. The complexity arises from fitting the GFKM with an $AR(1)$ structure, which models dependence only between adjacent pathways. While this structure captures local sequential correlation, it may not represent broader patterns of interaction. An extension to an $AR(\ell )$ structure, allowing each pathway to depend on multiple neighbors, provides a more flexible framework for modeling plausible inter‐pathway relationships, such as clustered pathways contributing jointly to the response. Although larger $\ell$ increases computational demands, particularly for inverting ${\sum }_r$ due to multiple correlations among pathways, this extension offers promise for improving both interpretability and realism.

Further investigation is needed to develop strategies for pathway ordering that incorporate correlations among genes within pathways. One approach is to order pathways by maximum pairwise correlations, while another is to identify a core sequence with the strongest correlations and then extend the ordering sequentially outward from both ends. Comprehensive simulation studies will be required to assess the performance and robustness of these approaches.

Author Contributions

PhilGeun Jin conducted all numerical analysis and wrote the manuscript, YoungHo Yun reviewed the manuscript, and Inyoung Kim developed a method and wrote and reviewed the manuscript.

Funding

The authors have nothing to report.

Conflicts of Interest

The authors declare no conflicts of interest.

Supporting information

Data S1. Supporting Information.

Appendix A. Full Conditional Distributions

The full conditional distributions for all parameters except for ${\rho }_m$ have closed forms. So, we draw the MCMC sample using Metropolis‐Hastings (MH) for ${\rho }_m$ and Gibbs sampling for all parameters except for ${\rho }_m$. The detailed forms of the conditional distributions for continuous and binary responses are summarized in below sections.

Full Conditional Distributions of Parameters for Continuous Response

Using our GFKM (1) and prior specification (2), a joint posterior distribution under continuous response is then

$$\begin{array}{cc}\hfill & \pi \left(\mathit{\beta },{\sigma }^2,\mathbf{\text{r}},{\mathit{\tau }}^2,{\mathit{\omega }}^2,{\lambda }_1^2,{\lambda }_2^2,\mathit{\rho }|\mathbf{\text{y}},\mathbf{\text{X}},\mathbf{\text{Z}}\right)\hfill \\ \hfill & \propto N\left\{\mathbf{\text{y}};\mathbf{\text{X}}\mathit{\beta }+\sum _{m=1}^M{\mathbf{\text{r}}}_m\left({\mathbf{\text{Z}}}_m\right),{\sigma }^2\mathbf{\text{I}}\right\}\hfill \\ \hfill & \times N\left(\mathit{\beta };{\mu }_{\beta },{\sigma }_{\beta }\right)\times \text{IG}\left({\sigma }^2;\mu ,\nu \right)\hfill \\ \hfill & \times \prod _{m=1}^M\text{Gamma}\left({\tau }_m^2;\frac{n+1}{2},\frac{{\lambda }_1^2}{2}\right)\hfill \\ \hfill & \times \prod _{m=1}^{M-1}\text{Gamma}\left({\omega }_m^2;1,\frac{{\lambda }_2^2}{2}\right)\hfill \\ \hfill & \times \prod _{h=1}^2\text{Gamma}\left({\lambda }_h^2;{\gamma }_h,{\delta }_h\right)\hfill \\ \hfill & \times \prod _{m=1}^M\text{Gamma}\left({\rho }_m;a_m,g_m\right).\hfill \end{array}$$

The full conditional distributions for parameters $(\mathit{\beta },{\sigma }^2,\mathbf{\text{r}},{\mathit{\tau }}^2,{\mathit{\omega }}^2,{\lambda }_1^2,{\lambda }_2^2)$ except for ${\rho }_m$ have closed forms because of Bayesian hierarchical structure and conjugate priors. The full conditional distribution of ${\rho }_m$ is

$$\begin{array}{cc}\hfill \left[{\rho }_m|-\right]& \propto \prod _{m=1}^M\frac{1}{|{\mathbf{\text{K}}}_m^{-1}(\mathit{\rho })|}\exp \left[-\frac{{\{{\mathbf{\text{r}}}_m^T{\mathbf{\text{K}}}_m^{-1}(\mathit{\rho }){\mathbf{\text{r}}}_m\}}^{1/2}}{2{\sigma }^2{\tau }_m^2}\right]\hfill \\ \hfill & \times \prod _{m=1}^M\{{\rho }_m^{a_m-1}{e}^{-g_m{\rho }_m}\}\text{for}m=1,\dots ,M,\hfill \end{array}$$

where $\mathit{\rho }=({\rho }_1,\dots ,{\rho }_M)$. When we draw MCMC samples from the full distribution for ${\rho }_m$ using MH, we found a very slow mixing, which results in a convergence problem. To overcome this issue, we adopt a restricted maximum likelihood (REML) and consider a REML‐based posterior distribution. Let ${\mathit{\theta }}_1$ denote the vector for the parameters of marginal covariance of Y for continuous response, that is, ${\mathit{\theta }}_1=({\sigma }^2,{\mathit{\tau }}^2,\mathit{\rho })$. The marginal covariance of Y for continuous response is then ${\sum }_R={\sigma }^2\mathbf{\text{I}}+{\sum }_{m=1}^M{\tau }_m^2{\mathbf{\text{K}}}_m$. The REML‐based posterior distribution of ${\rho }_m$ is

$$\begin{array}{cc}\hfill & \left[{\rho }_m|\mathit{\beta },{\sigma }^2,\mathbf{\text{r}},{\mathit{\tau }}^2,{\mathit{\omega }}^2,{\lambda }_1^2,{\lambda }_2^2,\mathbf{\text{y}},\mathbf{\text{X}},\mathbf{\text{Z}}\right]\hfill \\ \hfill & \propto \exp \left\{-\frac{1}{2}\log |{\sum }_R\left({\mathit{\theta }}_1\right)|\right.\hfill \\ \hfill & -\frac{1}{2}\log |{\mathbf{\text{X}}}^T{{\sum }_R}^{-1}\left({\mathit{\theta }}_1\right)\mathbf{\text{X}}|\hfill \\ \hfill & \left.-\frac{1}{2}{(\mathbf{\text{y}}-\mathbf{\text{X}}\mathit{\beta })}^T{{\sum }_R}^{-1}\left({\mathit{\theta }}_1\right)(\mathbf{\text{y}}-\mathbf{\text{X}}\mathit{\beta })\right\}\hfill \\ \hfill & \times \text{Gamma}\left({\rho }_m;a_m,g_m\right).\hfill \end{array}$$

We sample $[{\rho }_m|-]$ from the MH algorithm. The proposal distribution for ${\rho }_m$ is $\text{Gamma}({\gamma }_m,{\delta }_m)$. We set $E({\rho }_m)={\gamma }_m\times {\delta }_m={\rho }_m^{(t)}$, and $Var({\rho }_m)={\gamma }_m\times {\delta }_m^2=1$.

We denote ${\sum }_{m=1}^M{\mathbf{\text{r}}}_m({\mathbf{\text{Z}}}_m)=({\mathbf{\text{1}}}_M^T\otimes {\mathbf{\text{I}}}_n)\mathbf{\text{r}}$, where ${\mathbf{\text{1}}}_M={(1,1,\dots ,1)}^T$ is a $M\times 1$ vector, $\otimes$ is kronecker product, and ${\mathbf{\text{I}}}_n$ is identity matrix.

The full conditional distributions for all parameters are derived as follows:

$$\begin{array}{cc}\hfill & \left[\mathit{\beta }|{\sigma }^2,\mathbf{\text{r}},\mathbf{\text{y}},\mathbf{\text{X}},\mathbf{\text{Z}}\right]\sim N\left\{{\left({\mathbf{\text{X}}}^T\mathbf{\text{X}}\right)}^{-1}{\mathbf{\text{X}}}^T\right.\hfill \\ \hfill & \left.\left\{\mathbf{\text{y}}-({\mathbf{\text{1}}}_M^T\otimes {\mathbf{\text{I}}}_n)\mathbf{\text{r}}\right\},{\sigma }^2{({\mathbf{\text{X}}}^T\mathbf{\text{X}})}^{-1}\right\};\hfill \\ \hfill & \left[{\sigma }^2|\mathit{\beta },\mathbf{\text{r}},{\mathit{\tau }}^2,{\mathit{\omega }}^2,\mathbf{\text{y}},\mathbf{\text{X}},\mathbf{\text{Z}}\right]\sim \text{IG}\left[\frac{n(M+1)}{2}+\mu ,\right.\hfill \\ \hfill & \left.\frac{\begin{array}{c}{\left\{\mathbf{\text{y}}-\left({\mathbf{\text{1}}}_M^T\otimes {\mathbf{\text{I}}}_n\right)\mathbf{\text{r}}-\mathbf{\text{X}}\mathit{\beta }\right\}}^T\\ \left\{\mathbf{\text{y}}-({\mathbf{\text{1}}}_M^T\otimes {\mathbf{\text{I}}}_n)\mathbf{\text{r}}-\mathbf{\text{X}}\mathit{\beta }\right\}+{\mathbf{\text{r}}}^T{\sum }_{\mathbf{\text{r}}}^{-1}\mathbf{\text{r}}\end{array}}{2}+\nu \right];\hfill \\ \hfill & \left[\mathbf{\text{r}}|\mathit{\beta },{\sigma }^2,{\mathit{\tau }}^2,{\mathit{\omega }}^2,\mathbf{\text{y}},\mathbf{\text{X}},\mathbf{\text{Z}}\right]\sim N\left\{{\left(J_M\otimes {\mathbf{\text{I}}}_n+{\sum }_{\mathbf{\text{r}}}^{-1}\right)}^{-1}\right.\hfill \\ \hfill & \left.\left({\mathbf{\text{1}}}_M\otimes {\mathbf{\text{I}}}_n\right)(\mathbf{\text{y}}-\mathbf{\text{X}}\mathit{\beta }),{\sigma }^2{\left(J_M\otimes {\mathbf{\text{I}}}_n+{\sum }_{\mathbf{\text{r}}}^{-1}\right)}^{-1}\right\};\hfill \\ \hfill & \left[\frac{1}{{\tau }_m^2}|{\sigma }^2,\mathbf{\text{r}},{\lambda }_1^2,\mathbf{\text{y}},\mathbf{\text{X}},\mathbf{\text{Z}}\right]\sim \text{InvG}\left(\sqrt{\frac{{\sigma }^2{\lambda }_1^2}{||{\mathbf{\text{r}}}_m|{|}_{K_m}^2}},{\lambda }_1^2\right);\hfill \\ \hfill & \left[\frac{1}{{\omega }_m^2}|{\sigma }^2,\mathbf{\text{r}},{\lambda }_2^2,\mathbf{\text{y}},\mathbf{\text{X}},\mathbf{\text{Z}}\right]\sim \text{InvG}\left\{\sqrt{\frac{{\sigma }^2{\lambda }_2^2}{{\sum }_{i=1}^n{(r_{i,m+1}-r_{i,m})}^2}},{\lambda }_2^2\right\};\hfill \\ \hfill & \left[{\lambda }_1^2|{\mathit{\tau }}^2,\mathbf{\text{y}},\mathbf{\text{X}},\mathbf{\text{Z}}\right]\sim \text{Gamma}\left\{\frac{M(n+1)}{2}+{\gamma }_1,\sum _{m=1}^M\frac{{\tau }_m^2}{2}+{\delta }_1\right\};\hfill \\ \hfill & \left[{\lambda }_2^2|{\mathit{\omega }}^2,\mathbf{\text{y}},\mathbf{\text{X}},\mathbf{\text{Z}}\right]\sim \text{Gamma}\left(M-1+{\gamma }_2,\sum _{m=1}^{M-1}\frac{{\omega }_m^2}{2}+{\delta }_2\right);\hfill \\ \hfill & \left[{\rho }_m|\mathit{\beta },\mathbf{\text{r}},\mathbf{\text{y}},\mathbf{\text{X}},\mathbf{\text{Z}}\right]\propto \exp \left\{-\frac{1}{2}\log |{\sum }_R({\mathit{\theta }}_1)|\right.\hfill \\ \hfill & \left.-\frac{1}{2}\log |{\mathbf{\text{X}}}^T{{\sum }_R}^{-1}\left({\mathit{\theta }}_1\right)\mathbf{\text{X}}|-\frac{1}{2}{(\mathbf{\text{y}}-\mathbf{\text{X}}\mathit{\beta })}^T{{\sum }_R}^{-1}\left({\mathit{\theta }}_1\right)(\mathbf{\text{y}}-\mathbf{\text{X}}\mathit{\beta })\right\}\hfill \\ \hfill & \times \text{Gamma}({\rho }_m;a_m,g_m),\text{for}m=1,\dots ,M,\hfill \end{array}$$

where $J_M$ is an $M\times M$ matrix with all‐ones components, and InvG denotes the inverse Gaussian distribution.

Full Conditional Distributions of Parameters for Binary Response

Using our model described in Section 2.2, we can have the joint posterior distribution for a binary response as follows,

$$\begin{array}{cc}\hfill & \pi \left({\mathbf{\text{Y}}}^{\ast },\mathit{\beta },\mathbf{\text{r}},{\mathit{\tau }}^2,{\mathit{\omega }}^2,{\lambda }_1^2,{\lambda }_2^2,\mathit{\rho }|\mathbf{\text{y}},\mathbf{\text{X}},\mathbf{\text{Z}}\right)\hfill \\ \hfill & \propto \prod _{i=1}^n\left\{\mathbb{1}\left(Y_i^{\ast }>0\right)\mathbb{1}\left(y_i=1\right)+\mathbb{1}\left(Y_i^{\ast }\le 0\right)\mathbb{1}\left(y_i=0\right)\right\}\hfill \\ \hfill & \times \prod _{i=1}^{n}N\left(Y_i^{\ast };{\mathbf{\text{x}}}_i^T\mathit{\beta }+\sum _m{\mathrm{r}}_{i,m},1\right)\times N\left(\mathit{\beta };{\mu }_{\beta },{\sigma }_{\beta }\right)\hfill \\ \hfill & \times \prod _{m=1}^M\text{Gamma}\left({\tau }_m^2;\frac{n+1}{2},\frac{{\lambda }_1^2}{2}\right)\hfill \\ \hfill & \times \prod _{m=1}^{M-1}\text{Gamma}\left({\omega }_m^2;1,\frac{{\lambda }_2^2}{2}\right)\hfill \\ \hfill & \times \prod _{h=1}\text{Gamma}\left({\lambda }_h^2;{\gamma }_h,{\delta }_h\right)\hfill \\ \hfill & \times \prod _{m=1}^M\text{Gamma}\left({\rho }_m;a_m,g_m\right).\hfill \end{array}$$

where $\mathbb{1}(A)=1$ if $A$ is true and $\mathbb{1}(A)=0$ otherwise, which is indicator function. The full conditional distributions for parameters $({\mathbf{\text{Y}}}^{\ast },\mathit{\beta },\mathbf{\text{r}},{\mathit{\tau }}^2,{\mathit{\omega }}^2,{\lambda }_1^2,{\lambda }_2^2)$ except for ${\rho }_m$ have closed forms because of Bayesian hierarchical structure and conjugate priors. For sampling ${\rho }_m$, we also use the REML‐based posterior distribution of ${\rho }_m$. As we described in Appendix A, a REML‐based posterior distribution for binary response is then

$$\begin{array}{cc}\hfill & \left[{\rho }_m|{\mathbf{\text{Y}}}^{\ast },\mathit{\beta },\mathbf{\text{r}},{\mathit{\tau }}^2,{\mathit{\omega }}^2,{\lambda }_1^2,{\lambda }_2^2,\mathbf{\text{y}},\mathbf{\text{X}},\mathbf{\text{Z}}\right]\hfill \\ \hfill & \propto \exp \left\{-\frac{1}{2}\log \left|{\sum }_R\left({\mathit{\theta }}_2\right)\right|-\frac{1}{2}\log \left|{\mathbf{\text{X}}}^T{{\sum }_R}^{-1}\left({\mathit{\theta }}_2\right)\mathbf{\text{X}}\right|\right.\hfill \\ \hfill & \left.-\frac{1}{2}{\left({\mathbf{\text{Y}}}^{\ast }-\mathbf{\text{X}}\mathit{\beta }\right)}^T{{\sum }_R}^{-1}\left({\mathit{\theta }}_2\right)\left({\mathbf{\text{Y}}}^{\ast }-\mathbf{\text{X}}\mathit{\beta }\right)\right\}\hfill \\ \hfill & \times \text{Gamma}\left({\rho }_m;a_m,g_m\right).\hfill \end{array}$$

The proposal distribution for ${\rho }_m^{\ast }$ is the same as Appendix A. The full conditional distributions for binary responses are derived as follows:

$$\begin{array}{cc}\hfill & \left[Y_i^{\ast }|\mathit{\beta },\mathbf{\text{r}},\mathbf{\text{y}},\mathbf{\text{X}},\mathbf{\text{Z}}\right]\sim \left\{\mathbb{1}\left(Y_i^{\ast }>0\right)\mathbb{1}\left(y_i=1\right)\right\}N\hfill \\ \hfill & \left\{X\mathit{\beta }+\left({\mathbf{\text{1}}}_M^T\otimes {\mathbf{\text{I}}}_n\right)\mathbf{\text{r}},1\right\};\hfill \\ \hfill & \left[Y_i^{\ast }|\mathit{\beta },\mathbf{\text{r}},\mathbf{\text{y}},\mathbf{\text{X}},\mathbf{\text{Z}}\right]\sim \left\{\mathbb{1}\left(Y_i^{\ast }\le 0\right)\mathbb{1}\left(y_i=0\right)\right\}N\hfill \\ \hfill & \left\{X\mathit{\beta }+\left({\mathbf{\text{1}}}_M^T\otimes {\mathbf{\text{I}}}_n\right)\mathbf{\text{r}},1\right\};\hfill \\ \hfill & \left[\mathit{\beta }|{\mathbf{\text{Y}}}^{\ast },\mathbf{\text{r}},\mathbf{\text{y}},\mathbf{\text{X}},\mathbf{\text{Z}}\right]\sim N\left\{{\left({\mathbf{\text{X}}}^T\mathbf{\text{X}}\right)}^{-1}{\mathbf{\text{X}}}^T\right.\hfill \\ \hfill & \left.\left\{{\mathbf{\text{Y}}}^{\ast }-\left({\mathbf{\text{1}}}_M^T\otimes {\mathbf{\text{I}}}_n\right)\mathbf{\text{r}}\right\},{\sigma }^2{\left({\mathbf{\text{X}}}^T\mathbf{\text{X}}\right)}^{-1}\right\};\hfill \\ \hfill & \left[\mathbf{\text{r}}|{\mathbf{\text{Y}}}^{\ast },\mathit{\beta },{\mathit{\tau }}^2,{\mathit{\omega }}^2,\mathbf{\text{y}},\mathbf{\text{X}},\mathbf{\text{Z}}\right]\sim N\left\{{\left(J_M\otimes {\mathbf{\text{I}}}_n+{\sum }_{\mathbf{\text{r}}}^{-1}\right)}^{-1}\right.\hfill \\ \hfill & \left.\left({\mathbf{\text{1}}}_M\otimes {\mathbf{\text{I}}}_n\right)\left({\mathbf{\text{Y}}}^{\ast }-\mathbf{\text{X}}\mathit{\beta }\right),{\sigma }^2{\left(J_M\otimes {\mathbf{\text{I}}}_n+{\sum }_{\mathbf{\text{r}}}^{-1}\right)}^{-1}\right\};\hfill \\ \hfill & \left[\frac{1}{{\tau }_m^2}|\mathbf{\text{r}},{\lambda }_1^2,\mathbf{\text{y}},\mathbf{\text{X}},\mathbf{\text{Z}}\right]\sim \text{InvG}\left(\sqrt{\frac{{\lambda }_1^2}{||{\mathbf{\text{r}}}_m|{|}_{K_m}^2}},{\lambda }_1^2\right);\hfill \\ \hfill & \left[\frac{1}{{\omega }_m^2}|\mathbf{\text{r}},{\lambda }_2^2,\mathbf{\text{y}},\mathbf{\text{X}},\mathbf{\text{Z}}\right]\sim \text{InvG}\left\{\sqrt{\frac{{\lambda }_2^2}{{\sum }_{i=1}^n{\left(r_{i,m+1}-r_{i,m}\right)}^2}},{\lambda }_2^2\right\};\hfill \\ \hfill & \left[{\lambda }_1^2|{\mathit{\tau }}^2,\mathbf{\text{y}},\mathbf{\text{X}},\mathbf{\text{Z}}\right]\sim \text{Gamma}\left\{\frac{M(n+1)}{2}+{\gamma }_1,\sum _{m=1}^M\frac{{\tau }_m^2}{2}+{\delta }_1\right\};\hfill \\ \hfill & \left[{\lambda }_2^2|{\mathit{\omega }}^2,\mathbf{\text{y}},\mathbf{\text{X}},\mathbf{\text{Z}}\right]\sim \text{Gamma}\left(M-1+{\gamma }_2,\sum _{m=1}^{M-1}\frac{{\omega }_m}{2}+{\delta }_2\right);\hfill \\ \hfill & \left[{\rho }_m|{\mathbf{\text{Y}}}^{\ast },\mathit{\beta },\mathbf{\text{r}},\mathbf{\text{y}},\mathbf{\text{X}},\mathbf{\text{Z}}\right]\propto \exp \left\{-\frac{1}{2}\log |{\sum }_R\left({\mathit{\theta }}_2\right)|\right.\hfill \\ \hfill & -\frac{1}{2}\log |{\mathbf{\text{X}}}^T{{\sum }_R}^{-1}\left({\mathit{\theta }}_2\right)\mathbf{\text{X}}|\hfill \\ \hfill & \left.-\frac{1}{2}{\left({\mathbf{\text{Y}}}^{\ast }-\mathbf{\text{X}}\mathit{\beta }\right)}^T{{\sum }_R}^{-1}\left({\mathit{\theta }}_2\right)\left({\mathbf{\text{Y}}}^{\ast }-\mathbf{\text{X}}\mathit{\beta }\right)\right\}\hfill \\ \hfill & \times \text{Gamma}\left({\rho }_m;a_m,g_m\right)\text{for}k=1,\dots ,p,\hfill \end{array}$$

where the full conditional distributions of $Y_i^{\ast }$ are truncated standard normal distributions. The probit‐link is applied to implement Gibbs sampling for binary responses since full conditional distributions for all parameters except for ${\rho }_m$ have a closed form. These full conditional distributions are similar to those for continuous response by replacing $Y$ by $Y^{\ast }$ and ${\sigma }^2=1$.

Data Availability Statement

The data that support the findings of this study are openly available in BF‐GFKM at https://github.com/pgj439/BF‐GFKM.