Scalable Nonparametric Prescreening Method for Searching Higher-Order Genetic Interactions Underlying Quantitative Traits

The Gaussian process (GP) regression is theoretically capable of capturing higher-order gene-by-gene interactions important to trait variation non-exhaustively with high accuracy. Unfortunately, GP approach is scalable only for 100-200 genes and thus, not applicable for high...

Abstract

Gaussian process (GP)-based automatic relevance determination (ARD) is known to be an efficient technique for identifying determinants of gene-by-gene interactions important to trait variation. However, the estimation of GP models is feasible only for low-dimensional datasets (∼200 variables), which severely limits application of the GP-based ARD method for high-throughput sequencing data. In this paper, we provide a nonparametric prescreening method that preserves virtually all the major benefits of the GP-based ARD method and extends its scalability to the typical high-dimensional datasets used in practice. In several simulated test scenarios, the proposed method compared favorably with existing nonparametric dimension reduction/prescreening methods suitable for higher-order interaction searches. As a real-data example, the proposed method was applied to a high-throughput dataset downloaded from the cancer genome atlas (TCGA) with measured expression levels of 16,976 genes (after preprocessing) from patients diagnosed with acute myeloid leukemia.

RECENT biological literature indicates that robust identification of higher-order gene-by-gene interactions would provide explanations for a broad spectrum of biological/medical mysteries (e.g., Ernst et al. 2007; Phillips 2008; Wei et al. 2014; Taylor and Ehrenreich 2015; Ehrenreich 2017). In genomic association studies, gene-by-gene interactions are typically identified via exhaustive search by enumerating all possible pseudovariables (product variables) in the field of interest (see Wei et al. 2014). However, the number of pseudovariables becomes rapidly infeasible, imposing a serious computational challenge for typical genetic applications with high-throughput sequencing datasets (Cordell 2009; Wei et al. 2014).

Ideally, the dimension of a feature space could be reduced without enumerating all possible pseudovariables, producing fewer terms to be considered. However, a phenotype may exhibit strong gene-by-gene interaction effects in the presence of negligible main effects (Wei et al. 2014). Therefore, individual genes contributing multiplicatively to phenotypic variation may not be identifiable by linear prescreening/dimension reduction methods. Gaussian process (GP)-model-based automatic relevance determination (ARD) (Neal 1996; MacKay 1998; Rasmussen and Williams 2006) is known to be an efficient nonparametric technique for identifying determinants of gene-by-gene interactions, even when they do not exhibit marginal association with phenotype (Zou et al. 2010). It has actually been shown that the powered exponential kernels implicitly enumerate all possible interaction effects of individual covariates [a special case is shown in Jiang and Reif (2015)] and seem to provide a reasonable solution to the given dimension reduction problem. Nonetheless, a remarkable drawback of GP regression remains in the computational cost of learning an unknown covariance structure that is about $\mathcal{O}\left(n^3\right)$. Markov chain Monte Carlo (MCMC)-based implementation of GP-regression can handle ∼200 variables (Zou et al. 2010), and, despite some computational advancements, including a penalized maximum likelihood approach (Shi and Choi 2011; Savitsky et al. 2011; Yi et al. 2011; Moore et al. 2016), the applicable number of variables still remains in the hundreds.

In this paper, we propose a novel prescreening/dimension reduction method based on the powered exponential kernel to ensure the theoretical capability of finding determinants of gene-by-gene interactions up to any order. This forms a generalized Haseman–Elston (H–E) regression model (Haseman and Elston 1972) that preserves virtually all the major benefits of GP regression while extending scalability to datasets with tens of thousands of variables. Several related methods have been proposed in the kernel regression literature (e.g., Cover and Thomas 2006; Yamada et al. 2014). A so-called kernelized feature-wise Lasso proposed by Yamada et al. (2014) appears to be especially highly efficient, with the same scalability as our method. The major difference between our method and those approaches is that the method proposed in this paper coincides more with the ARD approach, in which the kernel parameters are estimated from data.

Moreover, the impact of linear main effects on nonlinear effect estimation is often overlooked in practice. Ignorance of this model hierarchy severely limits the full potential of actual dimension reduction techniques by markedly increasing the number of false positives. Thus, we provide an integrative multi-stage procedure in which the linear main effects are estimated in essence [used for exhaustive search in Kärkkäinen et al. (2015) and Mathew et al. (2018)]. The nonparametric part characterized by the powered exponential kernel is then used to explain the similarities of the estimated residual vector between individuals that is nearly orthogonal with respect to the hyperplane spanned by the linear effects.

This multi-stage framework potentially has a huge impact on a broad spectrum of biological/medical problems since the genetic basis of phenotypic regulation often contains interaction effects but also a considerable number of non-negligible main effects (e.g., Yeang and Jaakkola 2006; Maienschein-Cline et al. 2012; Ehrenreich 2017). As an illustration, we provide several simulation scenarios (low- and high-dimensional cases) with performance comparisons to other existing methods, including high-order interaction terms up to the fourth order. As a real data example, we identify two- and three-way interactive regulatory genes for the well known prognostic factor in acute myeloid leukemia (AML) using the cancer genome atlas (TCGA) dataset with measured expression levels of 16,975 genes/proteins after preprocessing. Without dimension reduction, this would yield 144,066,825 possible interaction pairs and >${10}^{11}$ possible three-way interactions terms, respectively. This implies that literature for the known three-way interactions is fairly sparse at the present time due to the small number of scalable methods; thus, all findings are likely to be novel (cf. Awad and Chen 2014).

However, verification of gene-by-gene interactions is extremely challenging and often half-heartedly imposed in practice (Wei et al. 2014). We provide a multi-stage verification procedure based on the guidance of Cordell (2009) and Wei et al. (2014) to substantiate identified interactions. First, true interaction terms are separated from those that appear to be significant due to the strong underlying main effects and unfavorable coexpressions between genes. Second, we require that the effects on phenotype of the remaining gene-by-gene interactions are respectively exposed in the same direction in another independent sample (see also Milne et al. 2008). Finally, the most promising findings are further inspected to see if they overlap with experimentally verified gene/protein information from curated databases such as STRING and Biogrid.

Materials and Methods

In a regular additive multilocus model, individual genes are assumed to be associated additively with the phenotypic variation modeled by linear regression models of the form

$$Y_i=\mu +{\displaystyle \sum _{j=1}^p{X}_{ij}{\beta }_j+{\epsilon }_i.}$$ (1)

In this context, $Y_i$ denotes the phenotype value of the ith observation, and $X_{ij}$ corresponds to the gene expression value of observation i at gene j, each of which is centered and standardized to have a zero mean and unit variance. Moreover, μ is the population intercept, whereas ${\beta }_j$ is the main effect of gene $j,$ and ${\epsilon }_i$ is the random error following a normal distribution $\mathcal{N}\left(0,{\sigma }^2\right)$.

Interaction regression model

The linear combinations of the main effects are often too limited to explain the phenotypic variation accurately. Hence, the effects of gene-by-gene interactions up to the order k $\left(k\ge 2\right)$ should be also included, which leads to extension of the main effect model (1):

$$Y_i=\mu +{\displaystyle \sum _{j=1}^p{X}_{ij}{\beta }_j+\dots +{\displaystyle \sum _{1\le t_1,\dots ,t_k\le p}{X}_{i{t}_1}\dots X_{i{t}_k}{\beta }_{t_1,\dots ,t_k}+{\epsilon }_i.}}$$ (2)

Here, ${\beta }_j$ represents the main effect for gene j, similar to model (1), and ${\beta }_{t_1,\cdots ,t_k}$ is the gene-by-gene interaction effect between genes $X_{i{t}_1},\dots ,X_{i{t}_k}.$ The random error ${\epsilon }_i$ is assumed to follow a normal distribution with a mean of zero and variance equal to ${\sigma }^2$. Note that this formulation includes all higher order multiplicative terms of individual genes $\left(X_j^2,X_j^3,\dots \right)$.

A short preface to the proposed strategy:

It is widely acknowledged that covariates which contribute multiplicatively to the phenotypic variation may not be identifiable marginally by linear association studies (see e.g., Cordell 2009; Mackay 2014). However, individual genes related to important gene-by-gene interactions are associated with the phenotype nonlinearly through an unknown link-function (see e.g., Sailer and Harms 2017). That is, each interaction relationship, for instance $Y=X_k{X}_l+\epsilon$ can be represented with the marginal phenotype-relationships for both genes $X_k$ and $X_l$ as

$$Y=f_k\left(X_k\right)+{\epsilon }_k\mathrm{and}Y=f_l\left(X_l\right)+{\epsilon }_l$$

with unknown nonlinear functions $f_k(\cdot ),$ $f_l(\cdot )$ and the corresponding marginal error terms ${\epsilon }_k$ and ${\epsilon }_l$. Nonlinear associations between phenotypic variation and interactive covariates could be therefore transformed to be linearly identifiable once the form of the transformation function is properly chosen. The proposed transformation is motivated by the Gaussian-kernel-based ARD (Neal 1996; MacKay 1998; Rasmussen and Williams 2006) due to its known capacity to identifying determinants of important gene-by-gene interactions from their nonlinear marginal effects (Zou et al. 2010). However, instead of estimating the unknown covariance structure for individuals, we propose a semiparametric model that explains the phenotypic similarity of individuals with the matched kernel entries. This can be seen as the H–E type regression model (Haseman and Elston 1972; Sham and Purcell 2001) upon which our framework is based.

Haseman–Elston regression model

The H–E regression method was introduced for linkage testing using sib-pair data. In the original H–E approach, the squared phenotypic difference of individuals was regressed on the proportion of alleles shared identical-by-descent at an observed marker locus. Generally, if the phenotype for the ith individual is $Y_i$, the H–E regression model can be represented as

$$f\left(Y_i,Y_j\right)=\mu +\beta {\Omega }_{i,j}+{\epsilon }_{i,j}$$ (3)

in which $f\left(Y_i,Y_j\right)$ represents an arbitrary measure of the phenotypic difference and ${\Omega }_{i,j}$ is an arbitrary measure of the genetic difference, respectively. Yet, the measure of genetic as well as phenotypic differences of a pair of individuals are often modeled in the regression framework as

$${\left(Y_i-Y_j\right)}^2=\mu +\beta \left(X_i-X_j\right)+{\epsilon }_{i,j}.$$ (4)

Alternative metrics between phenotype observations can be found, for instance, in the work of Sham and Purcell (2001). Some of these metrics, including the one used in this paper, are clearly not difference measures, but the models based on those measures are still referred as H–E regression models.

Note that when the H–E regression estimators are applied over the entire population for each possible pair of individuals, the datapoints can be dependent, conditional on predictor variables such that $Y_i-Y_h$ is correlated with $Y_i-Y_k$. However, these dependencies between the datapoints could be accounted for by additional random effect terms $u_{i,j}$ with an appropriate covariance structure, such that $u_{i,j}╨{\epsilon }_{i,j}$ and

$${\left(Y_i-Y_j\right)}^2=\mu +\beta \left(X_i-X_j\right)+u_{i,j}+{\epsilon }_{i,j}.$$ (5)

Moreover, the random effects $u_{i,j}$ are assumed to be independent from the residual terms ${\epsilon }_{i,j}$ (their distributional assumptions are discussed later).

Gaussian process model—powered exponential kernel

In the original GP regression, the estimation of an unknown function involves estimating the covariance structure ${\Sigma }_{ij}$ between individuals, which can be modeled, for instance, through the kernel function of the form

$$K\left(X_i,X_j;\rho \right)=\mathit{\varrho }\hspace{0.17em}\exp \left(-{\displaystyle \sum _{k=1}^p{\rho }_k{\Vert X_{ik}-X_{jk}\Vert }^{\gamma }}\right),$$ (6)

where $X\in {\mathrm{ℝ}}^p$ is the random vector ${\left[X_1,\dots ,X_p\right]}^T$ and $\mathit{\varrho },{\rho }_k$ are free parameters. This covariance function form is known as the powered exponential kernel (Neal 1996). Parameter ϱ regulates the magnitude of the exponential part, reflecting the variation in vertical scale, and $\gamma \in \left]0,2\right]$ regulates the covariance smoothness (see Shi and Choi 2011). The covariance function, especially for $\gamma =2,$ is the popular Gaussian kernel [e.g., used in Zou et al. (2010) and Bobb et al. (2015)]. However, the Gaussian kernel belongs to the class of infinitely differentiable functions and has been criticized for being overly smooth for many real-life applications (Rasmussen and Williams 2006). By using the powered exponential kernel instead, the covariance smoothness can be regulated by the $\gamma \in \left]0,2\right]$ parameter.

The inverse of ${\rho }_k$ is often called a bandwidth parameter. The bandwidth parameter $1/{\rho }_k$ represents the length scale characterizing how fast the covariance function is expected to vary along the axis of $X_k.$ The shorter the characteristic length of $X_k$, the faster the covariance function will vary as a function of $X_k$. The importance of each variable $X_k$ can therefore be determined by a variable specific bandwidth parameter $1/{\rho }_k$ or, equivalently, an inverse bandwidth parameter ${\rho }_k$. For example, ${\rho }_k\sim 0$ indicates that the covariance function is expected to be practically a constant function of variable $X_k$, which is therefore deemed irrelevant (Neal 1996; MacKay 1998; Rasmussen and Williams 2006). Large ${\rho }_k$ represents a shorter length scale, implying that variable $X_k$ is of high importance, respectively.

Representing higher-order interactions implicitly with kernels

In conjunction with the Gaussian process ARD (Neal 1996; MacKay 1998) we adopt a kernel-based nonlinear regression technique where a kernel function is used for implicitly transforming the original model matrix into a higher-dimensional Hilbert space. Let $\mathcal{X}\subset {\mathrm{ℝ}}^p$ be the sample space of the set of individual genes $X={\left[X_1,\dots ,X_p\right]}^T$. The set of all interaction terms in the model (2) can be considered as the projections of individual genes $X\in \mathcal{X}$ in a higher-dimensional feature space $\mathcal{F}\subset {\mathrm{ℝ}}^d$ defined by an explicit feature map

$${\phi }_X:{\mathcal{X}}_{\mathrm{\subset }{\mathrm{ℝ}}^p}\to {\mathcal{F}}_{\mathrm{\subset }{\mathrm{ℝ}}^d}$$ (7)

that enumerates all d possible interactions up to a specific order $l,$ where $1\le l\le p.$ The number of coordinates $d={\displaystyle {\sum }_{k=1}^l\left(\begin{array}{c}p\\ k\end{array}\right)+\left(l-1\right)p}$ in the mapping ${\phi }_X:\mathcal{X}\to \mathcal{F}$ therefore quickly becomes infeasible as a function of l imposing a serious computational challenge if the estimation proceeds explicitly in the interaction feature space $\mathcal{F}$.

However, it is known that (see e.g., Rasmussen and Williams 2006) for every positive semidefinite kernel function $K\left(\cdot ,\cdot ;\rho \right):\mathcal{X}\times \mathcal{X}\to \mathrm{ℝ},$ a Hilbert space $\mathrm{\mathscr{H}}$ and a mapping $\Phi \left(\cdot ;\rho \right):\mathcal{X}\to \mathrm{\mathscr{H}}$exist, such that, for all $X_i,X_j\in \mathcal{X},$ we have that

$$K\left(X_i,X_j;\rho \right)={\langle \Phi \left(X_i;\rho \right),\Phi \left(X_j;\rho \right)\rangle }_{\mathrm{\mathscr{H}}},$$ (8)

where ${\langle \cdot ,\cdot \rangle }_{\mathrm{\mathscr{H}}}$ is a norm in the Hilbert space $\mathrm{\mathscr{H}}.$ This implies that the mapping ${\phi }_X(\cdot )$ in (Equation 7) can be characterized by an appropriate kernel function $K\left(\cdot ,\cdot ;\rho \right),$ which implicitly computes the genetic similarity measure between $\Phi \left(X_i;\rho \right)$ and $\Phi \left(X_j;\rho \right)$ in the higher-dimensional feature space $\mathcal{F}\subset {\mathrm{ℝ}}^d$. The idea is to find a kernel function $K\left(\cdot ,\cdot ;\rho \right)$ such that the mapping $\Phi \left(\cdot ;\rho \right)$ would be as similar as possible to the feature map $\phi (\cdot )$ in (Equation 7).

Finding a suitable kernel provides huge computational alleviation, since $K\left(X_i,X_j;\rho \right)$ is much easier to compute than associated features $\Phi \left(X_i;\rho \right)$ and $\Phi \left(X_j;\rho \right).$ In the following sections we will apply identity (Equation 8) to reduce the number of individual genes in $X\in \mathcal{X}$ before mapping them to the interaction feature space $\mathcal{F},$ producing fewer interaction terms to be considered.

Powered exponential H–E regression model

Let us consider a mapping ${\Phi }_X:\mathcal{X}\to \mathcal{F}$ of individual genes X characterized by some positive semidefinite kernel $K_X\left(\cdot ,\cdot ;\rho \right),$ and a phenotypic similarity function $T_Y\left(\cdot ,\cdot \right):\mathcal{Y}\times \mathcal{Y}\to {\mathrm{ℝ}}_{\ge 0}$ with non-negative ranges, where $\mathcal{Y}$ is the sample space of the phenotype Y. We propose a generalized H–E regression model in which the conditional expectation of the phenotypic similarity given the pair $\left(X_i,X_j\right)\in \mathcal{X}\times \mathcal{X}$ is modeled by the inner product of $\Phi \left(X_i,\rho \right)$ and $\Phi \left(X_j,\rho \right)$ in $\mathcal{F}$ implicitly using the identity (Equation 8), such that

$$\mathbb{E}\left(T_Y\left(Y_i,Y_j\right)|\left(X_i,X_j\right)\right)=K_X\left(X_i,X_j;\rho \right).$$ (9)

In particular, the importance of each gene is characterized by the gene-wise kernel parameter vector $\rho \in {\mathrm{ℝ}}^p$ estimated from data as in the original ARD-approach (Neal 1996; MacKay 1998).

As we have defined $T_Y\left(\cdot ,\cdot \right):\mathcal{Y}\times \mathcal{Y}\to {\mathrm{ℝ}}_{+}$ to be a non-negative similarity measure, it is natural to adopt a multiplicative error model (see e.g., Eagleson and Muller 1997) suitable for positive responses around the mean $K_X\left(X_i,X_j;\rho \right)$, i.e.,

$$T_Y\left(Y_i,Y_j\right)=K_X\left(X_i,X_j;\rho \right){\epsilon }_{i,j},$$ (10)

where $\mathbb{E}\left({\epsilon }_{i,j}\right)=1$ and ${\epsilon }_{i,j}>0,$ with a probability of one for all pairs $\left(i,j\right)$ of individuals.

The phenotypic similarity function $T_{Y,\omega }\left(Y_i,Y_j\right)$ is defined in this paper as the inverse squared difference ${\left(Y_i-Y_j\right)}^{-2}$ if $\left|Y_i-Y_j\right|>\omega >0$ and ${\omega }^{-2}$ if $\left|Y_i-Y_j\right|<\omega$. For notational convenience, we assume that $P\left(\left|Y_i-Y_j\right|>0\right)=1$ when $i\ne j$ for a given dataset and set $\omega >0$ to be smaller than ${\min }_{i\ne j}\left|Y_i-Y_j\right|,$ implying that $T_{Y,\omega }\left(Y_i,Y_j\right)={\omega }^{-2}$ only when $i=j.$ Further, $K_X\left(\cdot ,\cdot ;\rho \right)$ was chosen to be the powered exponential kernel, which will be shown later to be well suited for finding higher-order interactions between gene-wise similarity measures in $\mathcal{F}$. This yields the proposed powered exponential Haseman-Elston (PH–E) model, written as

$$T_{Y,\omega }\left(Y_i,Y_j\right)=\mathit{\varrho }\hspace{0.17em}\exp \left(-{\displaystyle \sum _{k=1}^p{\rho }_k{\Vert X_{ik}-X_{jk}\Vert }^{\gamma }}\right){\epsilon }_{i,j},$$ (11)

where ${\epsilon }_{i,j}$ are assumed to be identically distributed positive error terms with $\mathbb{E}\left({\epsilon }_{i,j}\right)=1$ for all pairs $\left(i,j\right)$ of individuals. In this paper, the parameter $\gamma \in \left]0,2\right]$ regulating kernel smoothness is considered as fixed. However, the role of this parameter in model estimates is considered in the simulation section.

Dimension reduction—estimation of inverse bandwidth parameters

To estimate inverse bandwidth parameters $\rho \in {\mathrm{ℝ}}^p$ for dimension reduction, and to obtain an additive error term (see e.g., Eagleson and Muller 1997), we transform both sides of the model (10) with a strictly monotone increasing transformation function $\Psi :{\mathrm{ℝ}}_{+}\to \mathrm{ℝ},$ for which

$$\Psi \left(K_X\left(X_i,X_j;\rho \right){\epsilon }_{i,j}\right)=\Psi \left(K_X\left(X_i,X_j;\rho \right)\right)+\Psi \left({\epsilon }_{i,j}\right).$$ (12)

In this paper, we use a logarithmic-transformation function that transforms the PH–E model (11) into a regular multiple regression model over individual pairs $\left(i,j\right),$ where $i\ne j$ so that $T_{Y,\omega }\left(Y_i,Y_j\right)$ is always equal to ${\left(Y_i-Y_j\right)}^{-2}.$ The logarithmic-transformation function removes the exponential form of the right-hand-side and extricates the inverse bandwidth parameters i.e.,

$$\log \left({\left(Y_i-Y_j\right)}^{-2}\right)=\log (\mathit{\varrho })+{\displaystyle \sum _{k=1}^p\left(-{\rho }_k\right){\Vert X_{ik}-X_{jk}\Vert }^{\gamma }+\log \left({\epsilon }_{i,j}\right).}$$ (13)

In the light of the original PH–E model (11), the intercept $\log \left(\mathrm{\varrho }\right)$ can be interpreted as logarithmic magnitude parameter and $\log \left({\epsilon }_{i,j}\right)$ as the corresponding logarithmic error term. Here, the signs of the inverse bandwidth parameters ${\rho }_k$ $\left(k=1,\dots ,p\right)$ are written as negative for notational consistency with the standard linear models. Note that pseudo-observations $\log \left({\left(Y_i-Y_j\right)}^{-2}\right)$ and ${\Vert X_{ik}-X_{jk}\Vert }^{\gamma }$ for all possible individuals i and j $(i<j$ by the symmetry property of kernels) for some fixed $\gamma \in \left]0,2\right]$ are created beforehand, yielding a dataset with a sample size equal to $n\left(n-1\right)/2.$ The bandwidth parameters can now be estimated with linear estimation techniques by using pseudovariables $\log \left({\left(Y_i-Y_j\right)}^{-2}\right)$ and ${\Vert X_{ik}-X_{jk}\Vert }^{\gamma }$ in place of dependent and independent variables.

However, the problem with model (13) is that pseudo-observations are not mutually independent, as stated in the context of the model (5). To account for the dependencies between the pseudo-observations, one could include additional i.i.d. random effect terms $u_{i,j}$ in the model, such that $u_{i,j}╨\log \left({\epsilon }_{i,j}\right)$ and

$$\log \left({\left(Y_i-Y_j\right)}^{-2}\right)=\log (\varrho )+{\displaystyle \sum _{k=1}^p\left(-{\rho }_k\right){\Vert X_{ik}-X_{jk}\Vert }^{\gamma }+u_{i,j}+\log \left({\epsilon }_{i,j}\right).}$$ (14)

The random effect vector $u\in {\mathrm{ℝ}}^{n\left(n-1\right)/2}$ is assumed to follow a multivariate normal distribution $\mathcal{N}\left(0,{\sigma }^{2}G\right)$ where $G\in {\mathrm{ℝ}}^{n\left(n-1\right)/2\times n\left(n-1\right)/2}$ is a known expression covariance matrix between pseudo-observations. With reference to the methods where genomic relationship matrices are estimated from molecular markers located across the genome (see e.g., VanRaden 2008), matrix G could be estimated from the model matrix $Z\in {\mathrm{ℝ}}^{n\left(n-1\right)/2\times p}$ consisting of all pseudovariables ${\Vert X_{i1}-X_{j1}\Vert }^{\gamma },\dots ,{\Vert X_{ip}-X_{jp}\Vert }^{\gamma },$ such that

$$\widehat{G}=\left(Z-{\widehat{\mathit{z}}}_{\mathrm{row}}^T{1}_p\right){\left(Z-{\widehat{\mathit{z}}}_{\mathrm{row}}^T{1}_p\right)}^T/\left(p-1\right),$$

where ${\widehat{\mathit{z}}}_{\mathrm{row}}\in {\mathrm{ℝ}}^{n\left(n-1\right)/2}$ denotes a vector of the row means of the model matrix Z, and $1_p\in {\mathrm{ℝ}}^p$ is a constant vector of ones. In practice, the matrix G can be calculated simply, for instance in R as $\mathrm{cov}\left(t\left(Z\right)\right).$ The bandwidth parameters in model (14) can then be estimated with the typical linear mixed Lasso/elastic net type of estimators (Schelldorfer et al. 2011). However, it can be shown that results do not differ substantially between models (13) and (14). Especially in quantitative trait locus analyses, multilocus association models have been shown repeatedly to perform well without including any mixed model correction term to account for residual dependencies in the model (for example, Setakis et al. 2006; Pikkuhookana and Sillanpää 2009; Kärkkäinen and Sillanpää 2012; Würschum and Kraft 2015; Toosi et al. 2018). In particular, the proposed dimension reduction method produced almost identical results with and without including the random effect terms $u_{i,j}$ in the model in our simulations (see the Supplemental Materials A). Additionally, this mixed model version appears to be computationally very demanding (also shown in the Supplemental Materials A). Based on these facts, for simplicity, we decided to ignore the dependencies among the pseudo-observations in our example analyses.

Relaxed distributional assumptions for the random terms:

The distributional assumption $Y\sim \mathcal{N}\left(0,{\sigma }^{2}I\right)$ implies that ${\left(Y_i-Y_j\right)}^2$ follows a ${\chi }^2$-distribution ${\chi }^2\left(2\right)$ for $i\ne j.$ This yields relatively complex distributions for the random effect vector $u\in {\mathrm{ℝ}}^{n\left(n-1\right)/2}$ and the logarithmic residual vector $\log \left(\mathit{\epsilon }\right)\in {\mathrm{ℝ}}^{n\left(n-1\right)/2}$. However, it is shown that log-normal distributions offer appropriate approximations for ${\chi }^2$-distributions (Jouini et al. 2011). The dependent pseudovariables $\log \left({\left(Y_i-Y_j\right)}^{-2}\right)=-\log \left({\left(Y_i-Y_j\right)}^2\right)$ can be therefore treated approximately as normally distributed random variables, and, thus, it is adequate to assume that, for a known expression, covariance matrix G

$$u\sim \mathcal{N}\left(0,{\sigma }_u^{2}G\right)\mathrm{and\; log}\left(\mathit{\epsilon }\right)\sim \mathcal{N}\left(0,{\sigma }_{\epsilon }^{2}I\right)s.t.u╨\log \left(\mathit{\epsilon }\right).$$

Estimation with elastic net estimator:

In order to reduce the dimension of the interaction feature space $\mathcal{F}$ by removing multiplicatively nonimportant genes before enumeration, we assume that only a small number of the entries of $\left({\rho }_1,\dots ,{\rho }_p\right)$ are nonzero. The relative importance of each covariate with the sparsity assumption can be evaluated by penalized estimators. In this paper we use the elastic net estimator proposed by Zou and Hastie (2005), which solves the problem

$$\arg \underset{\mathit{\varrho },\rho }{\min }\left\{{\displaystyle \sum _{1\le i<j\le n}{(\log ({(Y_i-Y_j)}^{-2})-\log (\mathit{\varrho })-{\displaystyle \sum _{k=1}^p(-{\rho }_k){\Vert X_{ik}-X_{jk}\Vert }^{\gamma }})}^2}\right\}$$ (15)

$$\mathrm{subject\; to}(1-\alpha ){\Vert \rho \Vert }_2^2+\alpha {\Vert \rho \Vert }_1\le \lambda ,$$ (16)

where $\lambda \ge 0$ is the overall penalty-parameter. Relative proportion between the ${\ell }_1$-penalty and the ${\ell }_2$-penalty can be regulated by parameter $\alpha \in \left[0,1\right]$.

An explanation of why the proposed PH–E model is suitable for dimension reduction in higher-order interactions search

Let us define a H–E type interaction term over individuals i and j as a product of the ith and jth observations of a particular interaction term (e.g., $\left(X_{ik}{X}_{il}\right)\left(X_{jk}{X}_{jl}\right)$ for term $X_k{X}_l$). One beneficial feature of using the powered exponential kernel in Equation 11 is that it implicitly enumerates all higher-order H–E type interaction terms according to the following proposition:

Proposition 1:

The powered exponential kernel function $K_{\gamma }\left({\mathit{X}}_i,{\mathit{X}}_j;\rho \right)$ with $0<\gamma \le 2$ has the infinite series representation that contains all possible product terms ${\displaystyle {\prod }_{r\in {\mathrm{ℳ}}_s}{\rho }_r{\phi }_{{\mathrm{ℳ}}_s}\left({\mathit{X}}_i\right){\phi }_{{\mathrm{ℳ}}_s}\left({\mathit{X}}_j\right)}$ with respect to all possible subsets ${\mathrm{ℳ}}_s$ of indices $\left\{k_1,\dots ,k_s\right\}\subset \left\{1,\dots ,p\right\}$ of size s for all $1\le s\le p,$ where ${\phi }_{{\mathrm{ℳ}}_s}:{\mathrm{ℝ}}^p\to \mathrm{ℝ}$ is a mapping ${\phi }_{{\mathrm{ℳ}}_s}\left(X\right)={\displaystyle {\prod }_{k\in {\mathrm{ℳ}}_s}{X}_k.}$

Proof:

See Supplemental Materials A.

Moreover, phenotype Y and the interaction terms ${\displaystyle {\prod }_{k\in \mathrm{ℳ}}{X}_k}$ in model (2) are independent if, and only if, the proposed phenotypic similarity $T_Y\left(Y_i,Y_j\right)$ and the corresponding H–E type interaction terms ${\displaystyle {\prod }_{k\in \mathrm{ℳ}}{X}_{ik}}{\displaystyle {\prod }_{k\in \mathrm{ℳ}}{X}_{jk}}$ are independent:

Proposition 2:

Let us consider a random vector ${\mathit{X}}_i\in {\mathrm{ℝ}}^p,$ the corresponding phenotype $Y_i\in \mathrm{ℝ}$ and independent and identically distributed copies ${\mathit{X}}_j\in {\mathrm{ℝ}}^p$ and $Y_j\in \mathrm{ℝ}$. Then for any Borel-measurable function T we have that

$$Y_i╨{\displaystyle \prod _{k\in {\mathrm{ℳ}}_s}{X}_{ik}}\mathrm{if}\mathrm{and}\mathrm{only}\mathrm{if}T\left(Y_i-Y_j\right)╨\phi \left({\mathit{X}}_i\right)\phi \left({\mathit{X}}_j\right),$$ (17)

where $\phi :{\mathrm{ℝ}}^p\to \mathrm{ℝ}$ is a mapping $\phi \left(\mathit{X}\right)={\displaystyle {\prod }_{k\in {\mathrm{ℳ}}_s}{X}_k}$ for some fixed subset ${\mathrm{ℳ}}_s$ of indices $\left\{k_1,\dots ,k_s\right\}\subset \left\{1,\dots ,p\right\}$ of size $1\le s\le p.$

Proof:

See Supplemental Materials A.

Specifically, it can be shown that the individuals genes involved in phenotypically important interaction terms in the original model (2) are identifiable by the proposed PH–E method:

Proposition 3:

Let us consider an interaction model (2) with the random vector $\mathit{X}\in {\mathrm{ℝ}}^p$ and the phenotype $Y\in \mathrm{ℝ}.$ If we have that

$$\mathrm{Cov}\left(Y,{\displaystyle \prod _{k\in {\mathrm{ℳ}}_s}{X}_k}\right)\ne 0\mathrm{for}\mathrm{some}{\mathrm{ℳ}}_s=\left\{l_1,\dots ,l_s\right\}\subset \left\{1,\dots ,p\right\},$$

then the corresponding inverse bandwidth parameters ${\rho }_k$ $\left(k\in {\mathrm{ℳ}}_s\right)$ estimated by the PH-E method tend to be nonzero.

Proof:

See Supplemental Materials A.

A simple example:

Let us consider a simple three-way interaction model:

$$Y_i={\displaystyle \sum _{1\le k<l<m\le p}{\beta }_{klm}{X}_{ik}{X}_{il}{X}_{im}+{\epsilon }_i,\mathrm{where}i.i.d.{\epsilon }_i\sim \mathcal{N}\left(0,{\sigma }^2\right).}$$ (18)

Proposition 1 states that, by using the PH–E model (11) with the inverse squared phenotypic differences, we are implicitly assuming that

$$\mathbb{E}\left({\left(Y_i-Y_j\right)}^{-2}\right)={\displaystyle \sum _{1\le k<l<m\le p}{\rho }_k{\rho }_l{\rho }_m\left(X_{ik}{X}_{il}{X}_{im}\right)\left(X_{jk}{X}_{jl}{X}_{jm}\right).}$$ (19)

According to proposition 2, a particular H–E type interaction term $\left(X_{ik}{X}_{il}{X}_{im}\right)\left(X_{jk}{X}_{jl}{X}_{jm}\right)$ in model (19) is independent from the phenotypic similarity ${\left(Y_i-Y_j\right)}^{-2}$ if, and only if, $X_{ik}{X}_{il}{X}_{im}$ and $Y_i$ are independent in the original model (18). Proposition 3 further implies that the inverse bandwidth parameters ${\rho }_k,{\rho }_l,$ and ${\rho }_m$ estimated by the PH–E method tend to be nonzero if $X_{ik}{X}_{il}{X}_{im}$ and $Y_i$ are linearly dependent in the original model (18).

Multi-stage strategy for estimating bandwidth parameters

It is important to take into account that the main effects $\left({\beta }_1,\dots {\beta }_p\right)$ are identifiable primarily from marginal linear association studies. The interaction model (2) can therefore be portrayed implicitly through the following semiparametric regression model consisting of the explicitly parametrized main effects, i.e.,

$$Y=X\beta +g\left(X;\rho \right)+\epsilon ,\mathrm{where}\epsilon \sim \mathcal{N}\left(0,{\sigma }_0^{2}I\right).$$ (20)

Here, β is the main effect vector for a model matrix $X\in {\mathrm{ℝ}}^{n\times p}$, ε is an error term, and $g:{\mathrm{ℝ}}^{n\times p}\to {\mathrm{ℝ}}^{n\times p}$ is an arbitrary transformation function of the model matrix X. The parameter vector ρ displays the nonlinear importance of individual covariates through an unspecified nonparametric regression function.

In the presence of the strong main effects the proportion of phenotype variance explained by the nonparametric part may be less than required for the multiplicatively important covariates to be identifiable. Therefore, we recommend to use a multi-stage procedure in which the main effects are estimated beforehand, known as residual outcome analysis (see Kärkkäinen et al. 2015; Mathew et al. 2018). Then the nonparametric part $g\left(X;\rho \right)$ is used to explain the variation of the estimated residual vector $\widehat{\epsilon }=Y-\widehat{\beta }X$ yielding a nonparametric model

$$\widehat{\epsilon }=g\left(X;\rho \right)+{\epsilon }^{*},\mathrm{where}{\epsilon }^{*}\sim \mathcal{N}\left(0,{\sigma }_{*}^{2}I\right).$$ (21)

In this model, the parametric linear part affects the estimation of ρ less since the estimated residual vector $\widehat{\epsilon }=Y-\widehat{\beta }X$ is nearly orthogonal with respect to the hyperplane spanned by the variables with marginal effects.

However, a little caution is necessary regarding the residual outcome analysis. We refer to the paper of Demissie and Adrienne (2011), discussing the possible downward bias with respect to regression coefficients when residual outcome analysis is applied. The bandwidth parameters estimated from the model (21) are no longer interpretable on the scale of the original model (20). Yet, this approach increases the chances of interaction components/genes to be found correctly in the presence of the strong main effects. That is, the proportion of the variance in the residual vector $\widehat{\epsilon }$ predictable from the nonlinear part $g\left(X;\rho \right)$ is higher than the proportion of the variance in the phenotype Y explained by $g\left(X;\rho \right)$. Thus, the effects of the residual outcome analysis can be summarized as follows:

• Model coefficients: The inclusion of the residual step induces bias to the regression coefficients (Demissie and Adrienne 2011; Che et al. 2012).

• Interaction terms/components: The residual step can increase the number of correctly identified interaction terms/components in the model in the presense of the strong main effects. (Kärkkäinen et al. 2015; Mathew et al. 2018).

However, one should not consider this residual outcome analysis as a necessity but rather as an additional step to increase the signal-to-noise ratio of the nonlinear part in Equation 20. An illustrative example/comparison is given in the simulation section, and the inclusion of this step is made optional in the provided R-code.

Identifying interaction terms by residual outcome analysis

In this section, we provide a step-by-step guide to how to identify interaction effects in the presence of the strong main effects via the PH–E method with residual outcome analysis. Even though the same procedure can be applied to find three-way or higher-order interactions as well, for simplicity, we illustrate its usage via the following model, with main and two-way interaction effects:

$$Y_i={\beta }_0+{\displaystyle \sum _{m=1}^p{X}_{im}{\beta }_m+{\displaystyle \sum _{k\le l}{X}_{ik}{X}_{il}{\beta }_{kl}+{\epsilon }_i,}}$$ (22)

where ${\beta }_m$ is the main effect of the gene $X_m$ and ${\beta }_{kl}$ is the gene-by-gene interaction effect between genes $X_l$ and $X_k$, respectively. The random error ${\epsilon }_i$ is assumed to follow a normal distribution with a mean of zero and variance equal to ${\sigma }^2.$

Dimension reduction steps

Step 1. Computation of residuals:

The main effects ${\beta }_1,\dots ,{\beta }_p$ of the parametric part in Equation 20 are estimated by OLS-estimator if $n\ge p$ or by typical shrinkage methods (LASSO, elastic net etc.) if $n<p$. The estimated residual vector ${\left[{\widehat{\epsilon }}_1,\dots ,{\widehat{\epsilon }}_n\right]}^T$ where

$${\widehat{\epsilon }}_i=Y_i-{\widehat{\beta }}_0-{\displaystyle \sum _{m=1}^p{X}_{im}{\widehat{\beta }}_m}$$ (23)

is then nearly orthogonal with respect to the hyperplane spanned by the variables with the strongest marginal effects, and is thus considered as a new phenotype vector for the next step.

Step 2. Bandwidth estimation and dimension reduction:

Nonlinear effects are estimated by regressing the estimated logarithms of inverse squared residual differences between individuals i and j against the matched kernel entries by the PH–E model

$$\log \left({\left({\widehat{\epsilon }}_i-{\widehat{\epsilon }}_j\right)}^{-2}\right)=\log (\varrho )+{\displaystyle \sum _{k=1}^p\left(-{\rho }_k\right){\Vert X_{ik}-X_{jk}\Vert }^{\gamma }+\log \left({\epsilon }_{i,j}^{*}\right),}$$ (24)

where the logarithmic random error $\log \left({\epsilon }_{i,j}^{*}\right)$ of order two is assumed to follow a normal distribution with a mean of zero and variance equal to ${\sigma }_{*}^2.$

The relative importance of individual variables is evaluated by estimating variable specific inverse bandwidth parameters via the elastic net estimator (Eqs. 15–16). As in the original ARD (Neal 1996; MacKay 1998), we are interested in nonzero bandwidth parameters that are used to select the set of candidates $M=\left\{X_k|{\widehat{\rho }}_k\ne 0\right\}$ for further analysis. We suggest that penalty parameter λ in Equation 16 is chosen such that the requested number of candidates is obtained. The parameter α in Equation 16 depends more or less on the collinearity of the data, but the use of relative small values of α is recommended to avoid the common problems of LASSO (Zou and Hastie 2005).

Postdimension reduction steps

The set of chosen candidates $M=\left\{X_k|{\widehat{\rho }}_k\ne 0\right\}$ is then reanalyzed through a specific parametric model of interest, which, in the case of model (22) consists of all multiplicative terms of the second order. However, verification of gene-by-gene interactions is extremely challenging and often half-heartedly imposed in practice (Wei et al. 2014). Thus, we propose the following multi-stage verification procedure in conjunction with Wei et al. (2014):

Step 3. Removing pseudo-interaction terms:

Spurious interactions might be identified erroneously due to strong underlying main effects and unfavorable coexpression between genes (Wood et al. 2014; Wei et al. 2014). For instance, let us assume that gene $X_k$ and phenotype Y are strongly associated. Let us further assume that the expression-values of an another gene $X_l$ are scattered around one with small variance. This implies that the interaction term $X_k{X}_l$ (which is nearly equal to $X_k$) is also strongly associated with the phenotype Y, only because the gene $X_k$ has a strong marginal association with phenotype Y and is “unfavorable coexpressed” with gene $X_l.$ To remove such “pseudo-interactions”, all findings should be re-evaluated whether or not they exhibit non-negligible differences between the interaction and corresponding main effects. This can be done via a simple parametric interaction test in accordance with Aiken and West (1991): we test every possible pair $\left(X_k,X_l\right)$ from the set of chosen candidates M by regressing the original phenotype Y on both marginal variables and their interaction term, i.e.,

$$Y=X_k{\beta }_k+X_l{\beta }_l+X_k{X}_l{\beta }_{kl}.$$ (25)

Note that this step differs from the typical postselection significance tests. We are not yet making conclusions about the significance of the interaction effects in the light of the original model (22), and are therefore not accounting for possible selection bias (see e.g., Bühlmann and Mandozzi 2014). In this step we separate only the “right” interaction terms from those erroneously identified due to strong underlying main effects and unfavorable coexpression between genes (see Wood et al. 2014; Wei et al. 2014) as follows:

• Aiken–West decision rule for gene-by-gene interactions: If the interaction effect ${\beta }_{kl}$ in model (25) appears to be statistically more “significant” than the main effects ${\beta }_k$ and ${\beta }_l$, the test indicates that the interaction term can explain the phenotypic variation better than the individual covariates. When testing the null-hypothesis of zero-valued effects, the P-value corresponding to the interaction effect ${\beta }_{kl}$ should be relative small (e.g., ${10}^{-5}$) but also smaller than the P-values associated with the main effects. Only such interaction terms are kept for further consideration.

Respectively, to test a specific higher-order interaction term with the Aiken–West interaction test, we regress the phenotype on that interaction term and the corresponding lower-order interaction and marginal terms. The interaction effect of interest in this case should be statistically more “significant” than the corresponding main and lower-order interaction effects.

Step 4. Replication:

Wei et al. (2014) highlighted the importance of showing that the identified interaction effects are respectively exposed on the same direction on the phenotype in an independent dataset. We therefore recommend to retest (with the same Aiken–West test) remaining interaction terms from Step 3, or the whole analysis in some independent dataset. To some extent, this step also removes the false positives due to the selection bias caused by the prescreening step (Bühlmann and Mandozzi 2014). This is because the effects of the true underlying interaction terms are less sensitive to changes in data than those associated with false positive findings.

Exploring results from a biological point of view:

The remaining gene-by-gene interactions from steps 1–4 are inspected to see whether or not they have a biologically reasonable interpretation based on experimentally verified physical gene/protein interaction information. However, this should be considered only as an additional support for findings since it is totally dependent on existing biological knowledge about gene interactions, which is evolving constantly.

Data availability

The authors state that all data necessary for confirming the conclusions presented in the article are represented fully within the article. The proposed method is first evaluated with a normalized gene expression dataset used in the DREAM-challenge 9 (Noren et al. 2016), which is available upon registration (http://dreamchallenges.org/). These data were provided by Dr. Steven Kornblau from The University of Texas MD Anderson Cancer Center, and were obtained through Synapse syn2455683 as part of the AML DREAM-challenge. A high-throughput sequencing AML dataset downloaded from the cancer genome atlas (TCGA) is also analyzed, which is publicly accessible through the TCGA-website (https://cancergenome.nih.gov/). A list of genes used in our analyses is provided in Supplemental Materials B (genes that remained after preprocessing). However, we noted that the naming policies of several genes in this dataset have been recently changed, but the obsolete names can be linked to the updated ones, for example with Entrez gene IDs. For validation of the results, another AML cohort was also analyzed. This dataset was collected at Erasmus University Medical Center (Rotterdam) and is available for open research from the Gene Expression Omnibus (GEO, http://www.ncbi.nlm.nih.gov/geo, GDS4278, Taskesen et al. 2011). Supplemental material is available at figshare: https://doi.org/10.25386/genetics.9933239.

Results

We begin by illustrating the capability of the proposed method with a normalized gene expression dataset provided by DREAM-challenge (June 2014). This dataset (Noren et al. 2016) consists of measurements of 191 AML patients with measured expression levels of 231 genes and phosphoproteins probed by reverse phase protein array analysis, each of which follows a standard normal distribution. In the forthcoming section, new phenotypes are simulated conditionally on the real expression levels of these 231 genes.

PH–E method: one simulated three-way interaction effect without the main and lower-order interaction effects

In this section, we evaluate the efficiency of the proposed PH–E dimension reduction method, and illustrate the impact of changing the values of the kernel parameter γ on the model estimates. Performance was tested through a grid of different heritability levels with a simulated three-way interaction effect (dissembled at genes 21, 105, and 207 in the DREAM-challenge dataset where indexes start from the ACTB gene) without the main and lower-order interaction effects. Three different heritability levels were simulated, each involving 100 replicates of the phenotype vector. The population intercept was set to 0 for each replication, and, by varying normal residual variance, we obtained the averaged heritabilities 0.50, 0.30, 0.20 over replications.

The R-codes used (estimation was with the glmnet R-package, see Friedman et al. 2010) and simulated phenotype replicates are available in the Supplemental Materials such that the results can be easily replicated.

In this example, to produce comparable, but relatively sparse, results, we used the cross-validation (CV)-based value $3{\lambda }_{1se}$ for each test set. Here, ${\lambda }_{1se}$ denotes the largest value of λ such that error is within 1 SE of the minimum error (see the glmnet R-package). Note that the CV-based choice of an optimal penalty parameter λ is not recommended in real-data analysis since it tends to produce results that are too dense for our purposes. Instead, λ should be selected such that number of nonzero coefficients is less than a specific prefixed number (say 100). Moreover, the fixed proportion $\alpha =1/3$ between the ${\ell }_1$- and ${\ell }_2$-penalty terms in the elastic net estimator (Equation 15 and Equation 16) was used to estimate the bandwidth parameters.

Figure 1 displays the standardized estimates of the inverse bandwidth parameters ${\rho }_k$ for each variable $X_k,$ $k=1,\dots ,p$ averaged over 100 replications. The panel rows separate the estimates produced by different kernels parameters: $\gamma =2$ (the Gaussian kernel, panels C1–3), $\gamma =1$ (the exponential kernel, panels B1–3), and $\gamma =0.2$ (panels A1–3). For simulations corresponding to the heritability level of 0.50 (panels A1, B1, and C1), the averaged estimates of the bandwidth parameters associated with the correct genes (21, 105, 207) are much higher than irrelevant ones, regardless of the value of γ.

Figure 1.

Simulation analysis. Averaged estimates (over 100 replicates) of the inverse bandwidth parameters obtained by the PH–E method using distinct scenarios corresponding to the averaged heritability levels of 0.50 (A1, B1, and C1), 0.30 (A2, B2, and C2), and 0.20 (A3, B3, and C3), respectively. The panel rows separate the estimates produced by different kernels parameters: $\gamma =2$ (the Gaussian kernel, panels C1–3), $\gamma =1$ (the exponential kernel, panels B1–3) and $\gamma =0.2$ (panels A1–3). Black solid lines denote bandwidth parameter estimates and vertical red lines are the exact places of simulated phenotype associated genes. The CV-based penalty parameter value $3{\lambda }_{1se}$ was used for each scenario.

However, by decreasing the averaged heritability from 0.50 (panels A1, B1, and C1) to 0.30 (panels A2, B2, and C2) and further to 0.20 (panels A3, B3, and C3), the level of noise and the appearance of false positives start to depend on the kernel parameter γ. It appears that smaller values of γ systematically yield better results. Especially interesting is that probably the most widely used Gaussian kernel $\left(\gamma =2\right)$ shows relative poor performance compared to the exponential kernel, and even more so when compared to smaller values of λ.

Overall, all kernel parameter values have reasonable power to identify correct covariates with relative robust performance with respect to changes in heritability levels. However, simulations suggest that the performance of the PH-E method tends to be improved as the parameter γ decreases, based on which the value 0.2 of γ will be chosen in the real-data analysis section.

PH–E method via residual outcome analysis

The importance of the additional residual step was then evaluated by simulating the same three-way interaction term (the effect size is equal to 1) with three additional main effects (placed in 40, 150, and 180 with the effect sizes equal to 0.45, 0.45, and 0.30). In this case, only one level of heritability involving 100 replicates of the phenotype vector was simulated, with an averaged heritability of 0.40.

The inverse bandwidth parameters were estimated by the PH–E method, with and without the residual step presented in the Methods section. In both cases, the CV-based penalty parameter values $3{\lambda }_{1se}$ were used for each test set and replicate. Moreover, since the PH–E method with the residual step requires the main effects to be estimated beforehand, the elastic net estimator (with $\alpha =1/3$) was also used to estimate the main effects. The penalty parameter value corresponding to minimum prediction error selected by the CV-procedure was used. Then, the PH–E method was applied to the estimated residual vector in accordance with the Methods section.

Since the estimated residual vector is nearly orthogonal with respect to the hyperplane spanned by the main effects, the estimation of ρs should be now “enhanced” significantly. Without the residual step, the PH–E method was capable of finding the correct covariates (21, 105, 207), even though the main effects were inclined to be more exposed along with several false positives (Figure 2, left panel). As expected, by adjusting the PH–E method with the residual step, the estimates of bandwidth parameters associated with the simulated three-way interaction term are clearly separated from the main and other irrelevant effects (Figure 2, right panel).

Figure 2.

Simulation analysis. Panels represent averaged estimates of the bandwidth parameters obtained by the PH–E method without (left panel) and with (right panel) the additional residual step over 100 simulated replicates of phenotype with the averaged heritability level of 0.40. The CV-based penalty parameter value $3{\lambda }_{1se}$ was used in each scenario.

This yields significantly smaller search spaces after enumeration, even at this scale, regardless of the threshold by which candidates are chosen. For instance, in the right panel, the first false positive appears at the threshold value $\sim 0.22,$ whereas the number of nonzero coefficients in the left panel beyond that same hypothetical threshold point (black dotted vertical lines in Figure 2) is 16 (yielding 560 possible three-way combinations). Thus, based on this empirical evidence, the use of the PH–E method without the residual step severely prohibits its full potential.

Comparison to other nonparametric methods without the main effects

In this section, we compare the overall performance of the PH–E method to other general nonparametric dimension reduction/prescreening methods suitable for finding higher-order interactions. Common to all these methods is that they can completely avoid exhaustive search through the space of enumerated pseudovariables (gene-by-gene interactions) because the functional relationship between the corresponding explanatory variables and the phenotype is considered to be nonlinear. That is, each interaction relationship, for instance $Y=X_k{X}_l+\epsilon ,$ can be represented with the marginal phenotype-relationships for both genes $X_k$ and $X_l$ as

$$Y=f_k\left(X_k\right)+{\epsilon }_k\hspace{0.17em}\mathrm{and}\hspace{0.17em}Y=f_l\left(X_l\right)+{\epsilon }_l$$ (26)

with unknown nonlinear functions $f_k(\cdot ),$ $f_l(\cdot )$ and the corresponding marginal error terms ${\epsilon }_k$ and ${\epsilon }_l$.

The general nonparametric prescreening methods to be compared are (i) the mutual information (MI) approach (Frénay et al. 2013; Zhongxin et al. 2016), (ii) the distance correlation (DC)-based method (Li et al. 2012), and (iii) the HSIC-Lasso (Yamada et al. 2014). In the MI- and DC-approaches, one goes through all the gene expression values one at a time and measures the marginal phenotype relationships nonparametrically. A final outcome is a small subset of selected genes. In the HSIC-Lasso and the PH–E methods, a small subset of genes is selected from the transformed input variables (gene expressions) by the Lasso- or elastic net algorithm. Note that the MI method requires discretization of gene expression and trait values (which can be done in the corresponding packages). We used the infotheo R-package for the MI-method, energy R-package for the DC-method, and the publicly available Matlab-code for HSIC-Lasso (http://www.makotoyamada-ml.com/hsiclasso.html).

In the PH–E method, we use the powered exponential kernel (Equation 6) with the given kernel parameter $\gamma =\mathrm{0.2.}$ The performance is tested by simulating two three-way interactions and one two-way interaction (dissembled at genes 21, 105, 207, 41, 125, 225, 80, and 140 in the DREAM-challenge indexes starting from the ACTB gene) without the main effects, i.e.,

$$Y=X_{21}{X}_{105}{X}_{207}+X_{41}{X}_{125}{X}_{225}+X_{80}{X}_{140}+\epsilon ,$$ (27)

where $\epsilon \sim \mathcal{N}\left(0,{\sigma }^2\right).$ The effect sizes were set to be one for all interaction terms. We simulated 100 replicates of the phenotype vector such that the same phenotype replicates were analyzed by methods (i)–(iii). The population intercept was set to 0 for each replication, and normal residual variance ${\sigma }^2,$ such that we obtained an averaged heritability of 0.60 over replications. The R-code used for the simulations, as well as the simulated phenotype replicates, are available in the Supplemental Material.

Knowing the true simulated effects allows us to compare the receiver operating characteristics (ROC) between the MI-, DC-, HSIC-Lasso, and PH–E methods using the true positive rate (TPR), the false positive rate (FPR) and the area under the ROC-curve. Because the performance with a restricted, small FPR may be often particularly interesting, we also compare the partial areas under ROC-curves between methods truncated at FPR $\le \mathrm{0.2.}$

Since both the MI- and DC-based methods are unpenalized, we used relatively small penalty-parameter values $\left(\lambda ={10}^{-3}\right)$ for the PH–E method and HSIC-Lasso. Then, for the estimates ${\widehat{\rho }}_k\left(k=1,\dots ,p\right)$ obtained by each method, we shift the “decision” threshold value a through a grid $\left[\min \left(\widehat{\rho }\right),\max \left(\widehat{\rho }\right)\right]$ to produce the TPR and FPR for each a, such that

$$\mathrm{TPR}=\frac{\#\left(k:{\widehat{\rho }}_k>a\mathrm{and}{\rho }_k>a\right)}{\#\left(k:{\rho }_k>a\right)},$$ (28)

$$\mathrm{FPR}=\frac{\#\left(k:{\widehat{\rho }}_k>a\mathrm{and}{\rho }_k<a\right)}{\#\left(k:{\rho }_k<a\right)}.$$ (29)

Moreover, the fixed proportion $\alpha =1/3$ between the ${\ell }_1$- and ${\ell }_2$-penalty terms in the elastic net estimator (Equation 15 and Equation 16) was used to estimate the bandwidth parameters in the PH–E method. Figure 3 displays the ROC-curves for the MI-, DC-, HSIC-Lasso and PH–E methods constructed from the estimates averaged over 100 simulated phenotype replicates.

Figure 3.

Performance comparison: the ROC curves for the proposed PH-E method (black solid curve), HSIC-Lasso (red dashed curve), the DC-based method (blue dotted curve) and MI approach (gray dashdotted curve) characterized with respect to the estimates averaged over 100 simulated replicates. In the left panel (low-dimensional case), the dataset used consists of the measured expression levels of 231 genes (for 191 individuals), and, in the right panel (high-dimensional case), the measured expression levels of 16,976 genes (for 173 individuals), respectively.

Figure 3 (left panel) shows that both the PH–E method and HSIC-Lasso start to perform extremely well relatively quickly as the FPR increases, whereas the MI- and DC-based methods appear to be somewhat ineffective. The nontruncated areas under ROC-curves for the MI-, DC-, HSIC-Lasso, and PH–E methods were 0.614, 0.717, 0.831, and 0.934, respectively. While the proposed PH–E method consistently implies better accuracy than the other methods, the superiority of the PH–E method is particularly evident toward lower FPRs. The truncated areas under ROC-curves (truncated at FPR$\le 20\%$) for the MI-, DC-, HSIC-Lasso, and PH–E methods were 0.241, 0.330, 0.511, and 0.859, respectively. These results indicate that the PH–E method is more suitable approach for finding the components of higher-order interaction terms than its competitors.

Comparison to other nonparametric methods in high-dimensional settings with the main effects

In this section, the suitability of the PH-E method is illustrated for gene-expression datasets with a large number of features, and its accuracy once again compared to (i) the MI approach, (ii) the DC-based method, and (iii) the HSIC-Lasso. We use a high-throughput sequencing dataset downloaded from the cancer genome atlas (TCGA) including RNA-sequencing for 173 acute myeloid leukemia patients with measured expression levels of ∼20,000 genes publicly accessible from the TCGA website (https://cancergenome.nih.gov/).

This dataset was chosen due to its high dimensionality and multicollinearity among coregulated and coexpressed gene expression levels to evaluate the performance of methods (i)–(iii) in typical gene-expression datasets. Approximately $10\%$ of all pairwise correlations are >0.8. However, we removed genes with missing values, but also those with zero variance or very few unique expression values relative to the number of individuals. The latter is done by the nearZeroVar R-function (in the caret R-package). This yielded a dataset with 16,976 genes (for replication purposes, we provide a list of these 16,976 in Supplemental Materials B). Based on these remaining genes, we simulated new phenotypes such that;

$$Y=X_{6666}+0.2X_{9120}{X}_{2532}{X}_{8735}{X}_{1735}+0.3X_{4812}{X}_{5010}{X}_{10257}+0.2X_{754}{X}_{8132}+\epsilon ,$$

where $\epsilon \sim \mathcal{N}\left(0,{\sigma }^2\right).$ We simulated 100 replicates of the phenotype vector and analyzed the same phenotype replicates by the methods (i)–(iii). The normal residual variance ${\sigma }_{\epsilon }^2$ was set to be low $\left({\sigma }_{\epsilon }^2=0.2\right)$ due to the fact that the proportion of simulated interaction terms on explaining the overall phenotype variance is relatively small due to the large main effects.

The inverse bandwidth parameters in the model (11) were estimated by the PH–E method with the residual step presented in the methods section. Since the PH–E method with the residual step requires the main effects to be estimated beforehand, the elastic net estimator (with $\alpha =1$) was used to estimate the main effects. The penalty parameter value corresponding to minimum prediction error selected by the CV-procedure was used. Then, the PH–E method was applied to the estimated residual vector in accordance to the method section with $\alpha =1$ to provide sparsity and the given kernel parameter $\gamma =\mathrm{0.2.}$

In terms of computational time, all these methods were very similar, taking only roughly a few minutes to run the analysis for one replicate under the Windows operating system with a quad-core 3.40 GHz i7-processor (Intel 3770) and 16 GB of RAM. It can be seen in Figure 3 (right panel) that the PH–E method with the residual step performs evidently better than the MI- and DC-based methods over all FPRs. However, HSIC-Lasso cannot be compared in this dataset since it appeared to produce results that were too sparse for constructing ROC-curves regardless of the penalty parameter value used, and, therefore, were omitted from Figure 3. Nevertheless, the nontruncated areas under ROC-curves for the MI-, DC-, (HSIC-Lasso), and PH-E methods were 0.547, 0.456, (0.221), and 0.808, respectively. These simulated examples indicate that the proposed PH–E method is highly capable of finding components of higher-order interaction terms, even with challenging real-life high-throughput sequencing datasets.

Real data analysis

In this section, we identify interactive regulatory genes for known prognostic factors in AML. The main dataset was downloaded from the cancer genome atlas (TCGA), including RNA-sequencing for 173 AML patients with measured expression levels of ∼20,000 genes (median age is 47 years with range 18–72 years). Detailed descriptions of clinical and molecular characteristics with other information are publicly accessible from the TCGA website (https://cancergenome.nih.gov/). However, as in the simulation section, we removed genes with missing observations and those with zero variance or very few unique expression values relative to the number of individuals, yielding a dataset with 16,976 genes (a list of these genes is available in Supplemental Materials B).

Interaction regulation model for ARC/NOL3

The study by Mak et al. (2014) demonstrated that apoptosis repressor with caspase recruitment domain (ARC), encoded by the NOL3 gene [MIM: 605235], is repeatedly overexpressed in diagnosed AML samples, and is involved in the regulation of apoptosis of cardiac cells. However, the precise molecular mechanism regulating this antiapoptotic protein still remains unknown due to the overwhelming number of possible combinations in high-throughput datasets (Awad and Chen 2014). Here, the PH–E dimension reduction method was applied to identify two- and three-way regulatory interactions controlling the variation of ARC/NOL3 (1 of the remaining 16,976 genes).

Dimension reduction steps for finding interaction components

In accordance with the Methods section, the main effects were estimated with the elastic net estimator with $\alpha =0.2$ to obtain the residual vector of the first order. The penalty parameter λ was chosen by the CV criterion $\left(\lambda \approx 0.403\right),$ producing a set of nonzero main effects of size equal to $0.5\%$ of all genes (84 out of 16,975). Further consideration of the main effects will be omitted, whereas the major focus is on identifying and verifying two- and three-way interactions.

Whereas the variance of the NOL3 expression levels is approximately equal to one, the variance of the estimated residual vector $\widehat{\epsilon }$ of the first order is ∼0.16. The inverse bandwidth parameters are now estimated by the elastic net estimator (with $\alpha =1/3$) (Equation 15 and Equation 16) with respect to the estimated residual vector $\widehat{\epsilon }$ in accordance to step 2 presented in the Methods section. Based on the simulation studies, we used the powered exponential kernel, with a kernel parameter value =0.2. The penalty parameter λ was chosen to be the largest possible value, such that the number of nonzero inverse bandwidth parameters remains <2000, producing 1993 candidates from which the two-way interaction terms can be identified exhaustively, resulting a $\sim 99\%$ smaller interaction feature space than without the dimensional reduction step.

Note that larger penalty-parameter values are needed when higher-order interactions are considered. Once the estimated residual vector from the two-way interaction model was obtained, the PH–E method was further applied to find components involved in some three-way gene-by-gene interaction important to NOL3 variation. This was done by refitting the interaction model (2) with the main and two-way interaction effects enumerated only, over the previously found 1993 components. The residual vector ${\widehat{\epsilon }}^{*}$ of the second order can then be computed. Next, we repeated the dimension reduction step 2 with respect to the new residual vector ${\widehat{\epsilon }}^{*}$. In this case, we used the largest penalty parameter value λ, such that the number of nonzero bandwidth parameters was <200. Given 197 candidates from which the three-way interaction terms can be identified exhaustively yields 1,254,890 possible three-way interaction combinations to be screened exhaustively. At this point, the benefit of the proposed dimension reduction step is evident since after enumeration we have a interaction feature space which is ∼99.99985% smaller than the set we would get without the dimensional reduction step (>${10}^{11}$ enumerated candidates).

Postdimension reduction validation

In this section, we provide step-by-step guidance on how to analyze the set of remaining candidates from the previous steps through the postdimension reduction validation steps presented in the Methods section. We also provide biologically reasonable interpretation for two identified interaction terms as an example based on experimentally verified physical gene/protein interaction information. To that end, let us define the sets

$${\mathrm{ℳ}}_2=\left\{X_k{X}_l|{\widehat{\rho }}_k\ne 0\mathrm{and}{\widehat{\rho }}_l\ne 0,1\le k\le l\le p\right\}$$ (30)

$${\mathrm{ℳ}}_3=\left\{X_k{X}_l{X}_m|{\widehat{\rho }}_k*\ne 0,{\widehat{\rho }}_l^{*}\ne 0\mathrm{and}{\widehat{\rho }}_m*\ne 0,1\le k\le l\le m\le p\right\},$$ (31)

where ${\widehat{\rho }}_k$ and ${\widehat{\rho }}_l$ are the estimated inverse bandwidth parameters with respect to the residual vector $\widehat{\epsilon }$ of the first order. ${\widehat{\rho }}_k*$, ${\widehat{\rho }}_l^{*},$ and ${\widehat{\rho }}_m*$ are the estimated inverse bandwidth parameters with respect to the residual vector ${\widehat{\epsilon }}^{*}$ of the second order, respectively.

Parametric Aiken–West interaction test

We re-evaluated sets ${\mathrm{ℳ}}_2$ and ${\mathrm{ℳ}}_3$ to investigate whether or not the individual terms exhibit non-negligible differences between the interaction and corresponding lower-order interactions and marginal effects. In order to do so, we applied the parametric Aiken–West interaction test (Equation 23).

For the interaction terms in ${\mathrm{ℳ}}_2$ enumerated over the remaining candidates, we required that P-values for the interaction effect ${\beta }_{kl}$ in the Aiken–West test must be <${10}^{-5}$, and, for the corresponding main effects, ${\beta }_k$ and ${\beta }_l$ must be >${10}^{-3}.$ For the three-way interaction terms in ${\mathrm{ℳ}}_3,$ we required that P-values for the interaction effect ${\beta }_{klm}$ must be <${10}^{-5},$ and, for the corresponding lower-order interaction and main effects >${10}^{-3},$respectively. These requirements gave 1820 pairs out of 1,985,010 two-way interaction terms, i.e., removed ∼99.9% of all candidates from the set ${\mathrm{ℳ}}_2.$ Accordingly, the Aiken–West test produced only 185 three-way interactions out of 1,254,890 candidates (removed over $99.9\%$ of all three-way interactions from the set ${\mathrm{ℳ}}_3$).

Replication:

We further tested whether or not the effect found in the Aiken–West interaction test follows the same direction in terms of phenotype in an independent dataset. We therefore analyzed data from another AML cohort, which includes 156 de novo CN-AML patients (median age, 50 years, range: 16–77 years), collected at Erasmus University Medical Center (Rotterdam) between 1990 and 2008. All patients were treated uniformly under the study protocols of Dutch-Belgian Cooperative Trial Group for Hematology Oncology (HOVON, details of therapeutic protocol available at http://www.hovon.nl). All clinical, cytogenetic, and molecular information, as well as microarray data of these patients are publicly available at the Gene Expression Omnibus (GEO, http://www.ncbi.nlm.nih.gov/geo, GDS4278, Taskesen et al. 2011).

Interaction between HtrA serine peptidase 3 and protein phosphatase 1 regulatory subunit 15B

Based on our results, it is evident that the interaction of HtrA serine peptidase 3 (HTRA3 [MIM: 608785]) and protein phosphatase 1 regulatory subunit 15B [PPP1R15B (MIM: 613257)] exhibit stronger association with the variation in NOL3-expression levels [20.377, 95%-CI: (20.534, 20.221)] than the individual genes (Table 1, top panel). The effect of the same gene-by-gene interaction on NOL3 is also revealed as being in the same direction in an independent dataset [20.679, 95%-CI: (21.048, 20.310)] (Table 1, bottom panel).

Table 1. Listed two-way gene-by-gene interaction that is replicable in an independent dataset and biologically reasonable based on the current literature.

Gene name (TCGA)	Estimate	Std. Error	95% CI	P-value
HTRA3	0.201	0.072	[ 0.059, 0.344]	0.006
PPP1R15B	0.077	0.071	[–0.064, 0.218]	0.282
HTRA3: PPP1R15B	−0.377	0.079	[–0.534, −0.221]	$4.01\cdot {10}^{-6}$

Gene name (Omnibus)	Estimate	Std. Error	95% CI	P-value
HTRA3	0.024	0.085	[–0.144, 0.192]	0.781
PPP1R15B	0.047	0.194	[–0.336, 0.431]	0.807
HTRA3: PPP1R15B	−0.679	0.187	[–1.048, −0.310]	$3.8\cdot {10}^{-4}$

Estimated effect sizes, standard errors, 95% confidence intervals and P-values for the interaction and main effects on NOL3/ARC variation in the Aiken–West interaction test. These values are estimated separately from the main TCGA-dataset (top panel) and the independent Omnibus dataset (bottom panel).

Interpretation of findings based on the current literature and databases:

PPP1R15B belongs to the protein phosphatase (PP1) subfamily—a ubiquitous serine/threonine phosphatase family known to regulate cell division and apoptosis (Garcia et al. 2003). In particular, PP1 has been shown to repress tumor protein p53 activity through dephosphorylation (Lu et al. 2013), which can transcriptionally regulate the expression levels of our target ARC/NOL3 (Li et al. 2008).

Given recent evidence, HTRA3 may be essential for cancer development (see e.g., Zurawa-Janicka et al. 2012). HTRA3 is known to function as a tumor suppressor and inhibitor of the TGF-β signaling pathway. TGF-β in turn activates signaling pathways in a Smad-independent manner, such as mitogen-activating protein kinases (MAPKs) and phosphatidylinositol 3-kinase (PI3K)/Akt (Korrodi-Gregório et al. 2014). This provides reasonable a interpretation for our results since constitutive activation of both MAPK and PI3K/Akt pathways are common in AML and are shown to be key regulators of NOL3/ARC activity (Mak et al. 2014).

Three-way interactions

SPEN, PPP1R15B, and ZSWIM7:

The identified three-way interaction term between Spen family transcriptional repressor (SPEN [MIM: 613484]), PPP1R15B and Zinc Finger SWIM-Type containing 7 (ZSWIM7 [MIM: 614535]) also have stronger association with the variation of NOL3 expression levels [0.294, 95/% CI: (0.189, 0.400)] than the individual genes or their lower-order interaction terms (Table 2, top panel). Accordingly, the effect of the same gene-by-gene interaction on NOL3 expression has the same direction in another independent dataset [2.340, 95%-CI: (0.127, 0.4553)] with similar Aiken–West test results (Table 2, bottom panel).

Table 2. Listed three-way gene-by-gene interaction that is replicable in an independent dataset and biologically reasonable based on the current literature.

Gene name (TCGA)	Estimate	Std. Error	95% CI	P-value
SPEN	0.204	0.073	[ 0.070, 0.340]	0.003
PPP1R15B	0.194	0.068	[ 0.050, 0.339]	0.009
ZSWIM7	0.172	0.072	[ 0.049, 0.296]	0.007
SPEN: PPP1R15B	0.062	0.078	[−0.092, 0.216]	0.426
SPEN: ZSWIM7	−0.123	0.053	[−0.227, −0.019]	0.021
PPP1R15B: ZSWIM7	0.084	0.066	[−0.047, 0.215]	0.207
SPEN: PPP1R15B: ZSWIM7	0.294	0.053	[0.189, 0.400]	$1.3\cdot {10}^{-7}$

Gene name (Omnibus)	Estimate	Std. Error	95% CI	P-value
SPEN	−0.172	0.267	[−0.700, 0.355]	0.520
PPP1R15B	0.132	0.213	[−0.289, 0.554]	0.537
ZSWIM7	0.328	0.193	[−0.053, 0.711]	0.091
SPEN: PPP1R15B	−0.894	0.520	[−1.923, 0.135]	0.088
SPEN: ZSWIM7	−0.007	0.542	[−1.078, 1.065]	0.990
PPP1R15B: ZSWIM7	0.679	0.391	[−0.095, 1.453]	0.085
SPEN: PPP1R15B: ZSWIM7	2.340	1.120	[0.127, 4.553]	0.038

Estimated effect sizes, standard errors, 95% confidence intervals and P-values for the interaction and main effects on NOL3/ARC variation in the Aiken–West interaction test. These values are estimated separately from the main TCGA-dataset (top panel) and the independent Omnibus dataset (bottom panel).

Experimental studies by Oswald et al. (2002) have shown that SPEN is recruited by recombination signal binding protein for immunoglobulin kappa J (RBP-Jk) to a histone deacetylase (HDAC)-complex that is known to regulate the NOTCH signaling pathway. Further, a NOTCH-family member, NOTCH1 in turn regulates S-phase kinase associated protein 2 (SKP2) (Sarmento et al. 2005), which is associated directly with S-phase kinase associated protein 1 SKP1 in the SCF (Skp1/Cul1/F-box) protein complex. Finally, an experimental study of Sun et al. (2009) associated Skp1-Cul1-F-box complex to p53 regulation.

In the previous section, we explained that PPP1R15B is a member of the protein phosphatase (PP1) subfamily, which has been shown to repress p53 activity through dephosphorylation.

It is known that ZSWIM7 and SWSAP1 form a complex together (see Martino and Bernstein 2016), and their interplay between RAD51-paralogs has been discussed (Liu et al. 2011). Further, the RAD51C interacting protein XRCC3 has been shown to interact with FANCD2 protein in a newly identified complex (Somyajit et al. 2010), which in turn regulates the exonuclease CtIP (Yeo et al. 2014). Finally, CtIP binds directly to the SPEN repression domain (Oswald et al. 2005), building a bridge between SPEN and ZSWIM7.

Discussion

In this paper, we provide a fast multi-stage procedure for higher-order interaction search that makes two main contributions: (1) it extends the scalability of GP-based ARD (see Zou et al. 2010), while preserving its accuracy; and (2) it removes the influence of the main effects on the associated estimation process. Combined with the subsequent validation step, the proposed framework is therefore especially suitable for typical medical/biological problems with high-throughput gene/protein-expression sequencing datasets, e.g., identifying novel therapeutic targets and prognostic pathways for multi-factorial diseases. Moreover, the flexibility of the proposed dimension reduction method (PH–E) enables its extendability to a broad spectrum of important problems beyond higher-order interaction searches such as:

• Prediction of complex quantitative traits with the original GP-model: PH–E could be used to reduce the dimension of a feature space to achieve a smaller input space for the original GP-model, which is directly applicable for outcome prediction with small datasets.

• Identifying any kind of nonlinear effects: This framework could be used to find other nonlinear effects as well. This is relatively straightforward, since the parametric form can be defined user-specifically (see Step 3 in the Methods section).

Theoretical extension

Since we have considered only continuous explanatory variables (gene expression), applying the current framework for discrete explanatory variables (single nucleotide polymorphisms) requires further study. In this work, we have also completely adopted the frequentist point of view, whereas a natural advantage of the Bayesian approach would be to incorporate prior knowledge about each individual gene. However, since the model (13) is a regular linear model, applying Bayesian variable selection methods (O’Hara and Sillanpää 2009) instead of the elastic net estimator would be relatively straightforward.

Another issue omitted is the possible computational complexity caused by the increasing number of “observations” due to created pseudo-observations. If the resulting sample size becomes computationally overwhelming, one possible solution would be to consider only the tails of the phenotype distribution. Alternatively, one could perform random subsampling among observations and average the corresponding results. On the other hand, the creation of pseudo-observations might be an advantage if the sample size is extremely small, as stated implicitly by propositions 1 and 2.

The importance of validation

Use of the validation step was illustrated with the AML TCGA-dataset. During validation of our findings, we found that it was especially important to assure that the identified interaction effects on the phenotype are correspondingly exposed in the same direction in another independent dataset. For instance, a pairwise interaction between MAML1 and PPP1R12B was found to be strongly associated with NOL3 variation in the TCGA-dataset (P-value = $1.3\times {10}^{-5}$). According to curated databases, they also seem to be linked through an experimentally verified pathway structure. Together with the fact that the interaction between MAML1 and PPP1R15B appears to be statistically highly “significant” also in an independent omnibus dataset (P-value = 6.2$\times {10}^{-6}$), a conclusion could be easily drawn that this interaction truly is an important contributor to NOL3/ARC variation. However, the interaction effect in the Omnibus dataset is exposed in the opposite direction, causing contradictory interpretations.

In this study, replicable findings were also required to overlap with experimentally verified gene/protein interaction information downloaded from curated databases. However, this, by definition, precludes completely novel findings. This step should be generally considered only as additional support for findings rather than required criteria, since curated databases provide only recent biological information and are evolving constantly.