Information Metrics in Genetic Epidemiology

Abstract

Information-theoretic metrics have been proposed for studying gene-gene and gene-environment interactions in genetic epidemiology. Although these metrics have proven very promising, they are typically interpreted in the context of communications and information transmission, diminishing their tangibility for epidemiologists and statisticians.

In this paper, we clarify the interpretation of information-theoretic metrics. In particular, we develop the methods so that their relation to the global properties of probability models is made clear and contrast them with log-linear models for multinomial data. Hopefully, a better understanding of their properties and probabilistic implications will promote their acceptance and correct usage in genetic epidemiology. Our novel development also suggests new approaches to model search and computation.

1. Introduction

Information-theoretic metrics have been proposed for studying gene-gene and gene-environment interactions in genetic epidemiology (Sucheston et al., 2010); Chanda et al., 2007, 2008; Bush et al., 2008; Moore et al., 2006; Dong et al., 2008; Kang et al., 2008). The metrics considered in this paper been have shown to perform well in comparison to other methods in simulation experiments and case studies (Sucheston et al., 2010; Chanda et al., 2007, 2008). Although these metrics have proven very promising, they are typically interpreted in the context of communications and information transmission, diminishing their tangibility for epidemiologists and statisticians. Information concepts are typically defined as sums of entropy terms; such characterizations may be heuristically meaningful, but do not reveal their relevance to data and to global properties of probability models.

Cordell (2009) points out that the relationship of these approaches to more standard statistical methods is unclear. Clayton (2009) questions the usefulness of these methods compared to the logistic model for testing interaction and alternative methods which are based on conditional entropy. In this paper we clarify the interpretation of information-theoretic metrics, which have proven useful in the study of gene-gene and gene-environment interaction in genetic epidemiology. In particular, we develop the methods so that their relation to the global properties of probability models is made clear, and contrast them with log-linear models for multinomial data for further insight. We conclude that information-theoretic measures of statistical interaction have strong connections to more familiar approaches. However, they also differ in ways that may offer advantages and that encourage their further study. Hopefully, a better understanding of their properties and probabilistic implications will promote their acceptance as a promising approach for interaction detection in genetic epidemiology.

Section 2 of the paper, Background and Preliminaries, reviews fundamental concepts in information theory. The basic unit of information, the entropy of a distribution, is introduced. Kullback-Leibler divergence, Information concepts related to statistical independence, and key combinatorial tools are presented. Section 3 introduces and interprets information metrics specifically developed for detecting association in statistical genetic studies. Phenotype Associated Information (PAI) measures the association of a phenotype with a set of predictors, e.g. single nucleotide polymorphisms (SNPs). We describe it in statistical terms and give its asymptotic distribution and corresponding permutation test. Section 4 introduces the information metric of primary interest for the epidemiologic study of interaction, the K-way Interaction Index (KWII), and describes its properties in depth. Examples comparing KWII modeling with log-linear models for multinomial data give insight into the interpretation of KWII. The last of these examples demonstrates that KWII is a more parsimonious description of interaction. The relationship between KWII and PAI is made explicit, and a decomposition of entropy in terms of KWII analogous to components of variance is given. The section closes with a discussion of some alternative entropy-based methods. In section 5 we address model search algorithms, briefly describe previous work on the AMBIENCE algorithm (Chanda et al., 2008) for mining interactions using information metrics, and introduce a new algorithm. A discussion section concludes the paper.

2. Background and Preliminaries

2.1. Entropy

Entropies (Cover and Thomas, 2006) are the basic building blocks for the information metrics to be discussed. The Shannon entropy of a probability distribution with density f is defined to be H(f) = −E[log f (x)] and is described as representing “uncertainty” in x or “information gained” by observing a random variable x from the distribution f (x). Entropy is similar to the concept of variance. For a random variable X with density f we will often substitute the notation H(X) for H(f ). More generally, for a true distribution f, the entropy of a different density g is −E_f [log g(x)] = H(g) where the expectation is over the true distribution.

For a discrete distribution the information gained by observing an outcome x is defined to be –log p(x). Then the information expected from an experiment is:

$$H(X)=-\sum _{x}p(x)\text{log}\hspace{0.17em}p(x)=E[-\text{log}\hspace{0.17em}p(x)]$$

The capitalization of the argument of H(X) emphasizes that H depends on the distribution of the random variable X rather than an observed value, since it is an expectation. We adopt the convention that H(φ) = 0.

For a discrete Bernoulli p(x), the case p = 1 entails no uncertainty (and zero variance), so entropy is zero. In contrast, when outcomes are equiprobable, i.e., p = 0.5, the entropy is maximal because the uncertainty regarding the outcome is maximal. Note that H = 0 for both p = 0 and p = 1, so H is not a one-to-one transformation and it is a condensation of the detail in the probabilities.

2.2. Conditional entropy

Conditional entropy (Cover and Thomas, 2006) is defined in the natural way as:

$$H_X\hspace{0.17em}(Y)=E_x{E}_{y|x}[-\text{log}\hspace{0.17em}f(y|x)]$$

where E_y|x [−log f (y|x)], the expectation over the conditional distribution, is entropy calculated within the slice of data with a specific x value. Note that

$$\begin{array}{lll}{H}_X\hspace{0.17em}(Y)& =& E\hspace{0.17em}\left[-\text{log}\hspace{0.17em}\left(\frac{f\hspace{0.17em}(x,y)}{f\hspace{0.17em}(x)}\right)\right]\\ & =& H\hspace{0.17em}(X,Y)-H\hspace{0.17em}(X)\end{array}$$ (1)

This can be thought of as the entropy of y remaining after removing the entropy due to x, i.e., it is residual entropy. The joint entropy is the conditional entropy plus the entropy of the conditioning variable. Conditional entropy is distinguished from entropy of a conditional distribution H(Y|x) = E_y|x [−log f (y |x)], which is a function of x.

2.3. Kullback-Leibler divergence

Kullback-Leibler divergence (Anderson, 2008) is defined to be

$$I(f;q)=E_f[\text{log}\left(\frac{f(x)}{q(x)}\right)]$$

where f and q are probability density functions and the expectation is with respect to the true density f. We will use f and q to represent densities throughout this paper. Note that I(f ;q) = H(q) − H(f) so it can be viewed as comparing the entropies (with respect to f ) of the two density functions. Important properties of I(f ;q) include positive definiteness; i.e., I(f ;q) ≥ 0 and I(f ;q) = 0 if and only if f = q. Thus the unique density that minimizes the divergence is the true density function.

In our applications of K-L divergence q(x) will be a restriction of f (x) representing a statistical hypothesis. The K-L divergence therefore is the expected log-likelihood ratio.

2.4. Mutual information and statistical independence

The mutual information Inf (X ⊥ Y) (Cover and Thomas, 2006) describing the information in one random vector x about another random vector y is:

$$\mathit{\text{Inf}}(X\perp Y)=I(f(x,y);f(x)f(y))$$

This provides a measure of association or correlation between X and Y. In work with an information theory perspective, this correlation measure is denoted as “interaction”, a potential source of confusion for statistical applications; we will restrict the application of the term interaction to relations like nonadditive joint effects. If X and Y are independent, f (x, y) = f (x) f (y), so Inf (X ⊥ Y) = 0. Conversely, by the positive definiteness of divergence Inf (X ⊥ Y) = 0 only when X and Y are independent. Note that:

$$\begin{array}{lll}\mathit{\text{Inf}}\hspace{0.17em}(X\perp Y)& =& H\hspace{0.17em}(Y)+H\hspace{0.17em}(X)-H\hspace{0.17em}(X,Y)\\ & =& H\hspace{0.17em}(Y)-H_X\hspace{0.17em}(Y)\end{array}$$ (2)

by (1). Thus, if the uncertainty in Y remaining after accounting for X is large (small) as measured by H_X (Y) then the information X possesses about Y is small (large). Since Inf (X ⊥ Y) ≥ 0, H(Y) ≥ H_X (Y) and controlling for X can never increase entropy. The role of X here is analogous to that of a blocking variable in ANOVA or a covariate in regression analysis.

Conditional mutual information (Cover and Thomas, 2006) is defined in the natural way:

$$\begin{array}{lll}\mathit{\text{Inf}}(X\perp Y|Z)& =& I(f(x,y|z);f(x|z)f(y|z))\\ & =& E\left[\text{log}\left(\frac{f(x,y,z)f(z)}{f(x,z)f(y,z)}\right)\right]\end{array}$$

The expectation is over all the variables, so that Inf (X ⊥ Y |Z ) is not a function of Z.

Total correlation information (Chanda et al., 2007) is defined as

$$\mathit{\text{TCI}}\hspace{0.17em}(X_1,X_2,\dots ,X_K)=\sum _{i=1}^{K}H\hspace{0.17em}(X_i)-H\hspace{0.17em}(X_1,X_2,\dots X_K)$$ (3)

For example,

$$\mathit{\text{TCI}}(X_1,X_2,X_3)=H(X_1)+H(X_2)+H(X_3)-H(X_1,X_2,X_3)$$

This is easy to relate to divergence. Direct substitutions of the definitions of the entropies gives:

$$\begin{array}{lll}\mathit{\text{TCI}}(X_1,X_2,\dots ,X_K)& =& I(f(X_1,X_2,\dots ,X_K);{\prod }_{i=1}^{K}f(X_i))\\ & =& E[\text{log}\frac{f(X_1,X_2,\dots ,X_K)}{{\prod }_{i=1}^{K}f(X_i)}]\end{array}$$

For example,

$$\mathit{\text{TCI}}(X,Y,Z)=I(f_{x,y,z};f_x{f}_y{f}_z)$$

We see that TCI is the information against complete independence.

2.5. The Möbius transform

The Möbius transform is a key combinatorial tool in the development to follow. We will use it in section 4.1 to relate our interaction metrics to a factorization of the joint density. It will also be central to the algorithm given in section 5.2.

Let g and G be functions defined on the set of all subsets of a finite set Ω. Then the two statements i) g(x) = Σ_z⊆x G(z) and ii) G(z) = Σ_w⊆z(−1)^|z−w|g(w) are equivalent, where |·| is the cardinality of the argument. Thus a function defined by either expression can be inverted via the other equation of the pair. There is a Fast Möbius Transform (FMT, Thoma (1991)) for fast computation of the functions for all subsets of Ω in O(N2^N) time, where N = |Ω|:

To compute the Möbius transform using the FMT let initially

$$g_0(S)=G(S)$$

for all subsets S. Then iterate for j ranging over the variables in C as follows:

$$g_j(S)=\{\begin{array}{ll}{g}_{j-1}(S)\hfill & \text{if}\hspace{0.17em}j\notin S\hfill \\ g_{j-1}(S-\{j\})+g_{j-1}(S)\hfill & \text{if}\hspace{0.17em}j\in S\hfill \end{array}$$

The inverse transform is computed by initializing

$$G_0(S)=g(S)$$

for all subsets S. Then iterate for j ranging over the variables in C for as follows:

$$G_j(S)=\{\begin{array}{ll}{G}_{j-1}(S)\hfill & \text{if}\hspace{0.17em}j\notin S\hfill \\ -G_{j-1}(S-\{j\})+G_{j-1}(S)\hfill & \text{if}\hspace{0.17em}j\in S\hfill \end{array}$$

3. A measure of association: Phenotype Associated Information (PAI)

Since epidemiologic studies usually distinguish a response variable P (phenotype in our case) from the vector of predictors x, a concept PAI(X, P), the Phenotype Associated Information has been introduced (Chanda et al., 2007):

$$\mathit{\text{PAI}}(X,P)=\mathit{\text{TCI}}(P,X_1,X_2,\dots ,X_K)-\mathit{\text{TCI}}(X_1,X_2,\dots ,X_K)$$

This can be seen to be I(f_x,P; f_x f_P) = Inf (X ⊥ P). That is, PAI measures the association of P with the multivariate vector x. Note that PAI(X, P) = PAI(P, X) since Inf (X ⊥ P) = Inf (P ⊥ X). Using (1), PAI can be characterized as:

$$\begin{array}{lll}\mathit{\text{PAI}}\hspace{0.17em}(X,P)& =& H\hspace{0.17em}(X)+H\hspace{0.17em}(P)-H\hspace{0.17em}(X,P)\\ & =& H\hspace{0.17em}(P)-H_X\hspace{0.17em}(P)\end{array}$$ (4)

the reduction in the entropy of P due to X, i.e., the difference between the entropy of P and the residual entropy.

PAI(X, P) = 0 if and only if x and P are independent, so we can use properties of probabilistic independence to explicate the meaning of a null PAI. For example, by the Reduction Theorem (Whittaker, 1990), PAI(X, P) = 0 ⇔ P ⊥ X ⇒ P ⊥ X₁, P ⊥ X₁, . . . P ⊥ X_K, so each x_i alone contributes no information about P, i.e., no marginal effect. By the Block Independence Theorem (Whittaker, 1990), when the joint density of P and X is positive P ⊥ X ⇔ both P ⊥ X_A |X_B and P ⊥ X_B |X_A are true, where A, B is any partition of the variables in x. Thus the unique information regarding P possessed by any subset of the x’s is zero. For example, PAI(X, P) = 0 means that P ⊥ X_i |X − X_i, i.e., P is independent of each X_i given the remaining predictors. This is analogous to every regression coefficient being zero in a multiple prediction model regressing P on the combined set of predictors.

3.1. Statistical distribution of PAI

For multinomial data the sample estimate of PAI is obtained by substituting maximum likelihood estimates of the cell probabilities into the definition. The implication of this for statistical inference about PAI is given by Proposition 7.4.2 of Whittaker (1990):

Proposition: let p̂(x) be the unrestricted maximum likelihood estimate of the probability of cell x, i.e. the cell count n(x) divided by the number of observations N. A model M is defined to be a set of restrictions on the probabilities p. Let p̂^M(x) be the maximum likelihood estimate under M. The deviance of M is

$$\mathit{\text{dev}}(M)=2NI(\widehat{p},{\widehat{p}}^M)=2\sum _{x}n(x)\text{log}\frac{n(x)}{N{\widehat{p}}^M(x)},$$

which is asymptotically chi-squared with degrees of freedom equal to the number of parameters set to zero.

A model PAI(x, P) = 0 implies p̂^M = p̂_x p̂_P where p̂_x and p̂_P are unrestricted maximum estimates in the marginal tables of x and P. Since I(f_xP; f_x f_P) = Inf (X ⊥ P)=PAI(X, P), the empirical value of 2N PAI(X, P) has an asymptotic chi-squared distribution. The degrees of freedom is (r_P − 1)(Πr_Xi − 1), where r_A is the number of values the random variable A can take. The chi-squared distribution is only an approximation to the exact sampling distribution and is strictly valid only in large samples. Since the cell frequencies decrease as interaction order increases, larger sample sizes are required for testing interactions.

Since PAI(X, P)= Inf (X ⊥ P), when the sample size is insufficient to permit the reliable use of asymptotic methods a permutation method can be used. Randomly permuting the phenotype assignments will produce observations from the null distribution with X and P independent, yielding an empirical null distribution of PAI(X, P).

4. A information metric for interaction: K-Way Interaction Index (KWII)

Sucheston et al. (2010) and Chanda et al. (2007, 2008) have explored in detail the KWII, which is interpreted as an interaction of K variables. The KWII provides a consistent definition of statistical interaction which is of the same form for any choice of joint distribution of the variables.

The KWII of a set of variables X is defined as

$$\mathit{\text{KWII}}(X)=-\sum _{Z\subseteq X}{(-1)}^{|X-Z|}H(Z)$$

where H(φ) = 0.

It is clear from the definition that KWII is symmetric in its variables. The KWII of a single variable is negative entropy. For two variables it is mutual information (i.e. a measure of association/correlation). For three or more variables we will see that it captures the notion of statistical interaction. Note that KWII(X) is an expectation over the set of random variables X, not a function of an observed value x. A characterization of KWII is given by the following recursion:

Proposition: Define KWII_Y (X) = −Σ_Z⊆X(−1)^|X−Z|H_Y (Z). Then

$$\mathit{\text{KWII}}(X,Y)={\mathit{\text{KWII}}}_Y\hspace{0.17em}(X)-\mathit{\text{KWII}}(X),$$ (5)

where Y is a single variable, i.e. |Y| = 1, and X is a set of random variables not including Y.

Proof:

$$\begin{array}{l}\mathit{\text{KWII}}(X,Y)=-{\sum }_{Z\subseteq X\cup Y}{(-1)}^{|X\cup Y|-|Z|}H(Z),\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\text{by}\hspace{0.17em}\text{definition}.\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}=-{\sum }_{Z\subseteq X}{(-1)}^{|X|+|Y|-|Z|}H(Z)-{\sum }_{Z\subseteq X}{(-1)}^{|X|+1-|Z\cup Y|}H(Z,Y),\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\text{decomposing}\hspace{0.17em}\text{the}\hspace{0.17em}\text{summation}\hspace{0.17em}\text{into}\hspace{0.17em}\text{sets}\hspace{0.17em}\text{excluding}\hspace{0.17em}\text{Y}\hspace{0.17em}\text{and}\hspace{0.17em}\text{sets}\hspace{0.17em}\text{including}\hspace{0.17em}\text{Y}.\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}={\sum }_{Z\subseteq X}{(-1)}^{|X|-|Z|}H(Z)-{\sum }_{Z\subseteq X}{(-1)}^{|X|+1-|Z\cup Y|}H(Z,Y),\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\text{since}|Y|=1.\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}=-\mathit{\text{KWII}}(X)-{\sum }_{Z\subseteq X}{(-1)}^{|X|+1-|Z\cup Y|}(H_Y(Z)+H(Y)),\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\text{by}\hspace{0.17em}(1)\hspace{0.17em}\text{and}\hspace{0.17em}\text{the}\hspace{0.17em}\text{definition}\hspace{0.17em}\text{of}\hspace{0.17em}\mathit{\text{KWII}}.\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}=-\mathit{\text{KWII}}(X)-{\sum }_{Z\subseteq X}{(-1)}^{|X|-|Z|}{H}_Y(Z)-H(Y){\sum }_{Z\subseteq X}{(-1)}^{|X|-|Z|},\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\text{since}\hspace{0.17em}|Y|\hspace{0.17em}=1.\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}=-\mathit{\text{KWII}}(X)+{\mathit{\text{KWII}}}_Y(X)-H(Y){\sum }_{Z\subseteq X}{(-1)}^{|X|-|Z|}\end{array}$$

Note that

$$\sum _{Z\subseteq X}{(-1)}^{|X|-|Z|}=\sum _{i=0}^{|X|}\sum _{Z\subseteq X\wedge |Z|=i}{1}^{|Z|}{(-1)}^{|X|-|Z|}=\sum _{i=0}^{|X|}\left(\begin{array}{c}|X|\\ i\end{array}\right)1^{|Z|}{(-1)}^{|X|-|Z|}.$$

Thus the summation in the last term above is equal to (1 − 1)^|X| = 0 since it is of the form of the binomial theorem, ${\sum }_{i=0}^n\left(\begin{array}{c}n\\ i\end{array}\right)a^i{b}^{n-i}={(a+b)}^n$.

Note that (5) holds no matter which variable we condition on, i.e., KWII(X) = KWII_Xi(X \ X_i) − KWII(X \ X_i) for any i, where A \ B is the set difference, i.e. A with the elements shared with B removed.

The following lemma gives a method for computing KWII_Y (X).

Lemma: Let KWII(X|Y = y) be the value of KWII(X) calculated for the subset of observations taking value y for the random variable Y. Then KWII_Y (X) = E_yKWII(X|Y = y)

Proof:

$$\begin{array}{lll}{\mathit{\text{KWII}}}_Y(X)& =& -{\sum }_{Z\subseteq X}{(-1)}^{|X-Z|}{H}_Y(Z)\\ & =& -{\sum }_{Z\subseteq X}{(-1)}^{|X-Z|}{E}_y{E}_{x|y}[-\text{log}\hspace{0.17em}f(Z|y)]\\ & =& E_y\left[-{\sum }_{Z\subseteq X}{(-1)}^{|X-Z|}{E}_{x|y}[-\text{log}\hspace{0.17em}f(z|y)]\right]\\ & =& E_y\mathit{\text{KWII}}(X|Y=y).\end{array}$$

Thus if KWII(X) is computed within each subset of observations with a fixed y value, then KWII_Y (X) is obtained as the expectation of those KWII over the values of y.

Example 3.1. Consider the 2 x 2 tables $\begin{array}{ccc}2& 2& 4\\ 1& 2& 3\\ 3& 4& 7\end{array}$ and $\begin{array}{ccc}2& 2& 4\\ 2& 1& 3\\ 4& 3& 7\end{array}$.

We consider the two tables to be conditional on a third binary variable. For example, the row and column variables G_r and G_c might represent binary indicator variables corresponding to the dominance coding of the genotypes at two different loci and the tables might be stratified according to a binary phenotype P. For the left table (corresponding to P = 0) $H(G_r,G_c)=\frac{2}{7}\mathit{\text{log}}(\frac{2}{7})+\frac{2}{7}\mathit{\text{log}}(\frac{2}{7})+\frac{2}{7}\mathit{\text{log}}(\frac{2}{7})+\frac{1}{7}\mathit{\text{log}}(\frac{1}{7})=-1.35178$; ; $H(G_r)=\frac{4}{7}\mathit{\text{log}}(\frac{4}{7})+\frac{3}{7}\mathit{\text{log}}(\frac{3}{7})=-0.682908$; $H(G_c)=\frac{3}{7}\mathit{\text{log}}(\frac{3}{7})+\frac{4}{7}\mathit{\text{log}}(\frac{4}{7})=-0.682908$; H(φ) = 0. KWII(G_r, G_c|P = 0) for that table is −H(G_r, G_c) + H(G_r) + H(G_c) = −0.0140322. For the right table (P = 1) $H(G_r,G_c)=\frac{2}{7}\mathit{\text{log}}(\frac{2}{7})+\frac{2}{7}\mathit{\text{log}}(\frac{2}{7})+\frac{1}{7}\mathit{\text{log}}(\frac{1}{7})+\frac{2}{7}\mathit{\text{log}}(\frac{2}{7})$; $H(G_r)=\frac{4}{7}\text{log}(\frac{4}{7})+\frac{3}{7}\mathit{\text{log}}(\frac{3}{7})$; $H(G_c)=\frac{4}{7}\mathit{\text{log}}(\frac{4}{7})+\frac{3}{7}\mathit{\text{log}}(\frac{3}{7})$; H(φ) = 0. Since these entropies are the same as those of the left table KWII(G_r, G_c|P = 1) = KWII(G_r, G_c|P = 0) = −0.0140322.

The pooled table (representing the marginal distribution) is

$$\begin{array}{l}\begin{array}{l}\begin{array}{ccc}4& 4& 8\\ 3& 3& 6\\ 7& 7& 14\end{array}\hfill \end{array}\hfill \end{array}$$

In the margin $H(G_r,G_c)=\frac{4}{14}\mathit{\text{log}}(\frac{4}{14})+\frac{4}{14}\mathit{\text{log}}(\frac{4}{14})+\frac{3}{14}\mathit{\text{log}}(\frac{3}{14})+\frac{3}{14}\mathit{\text{log}}(\frac{3}{14})=-1.37606$; $H(G_r)=\frac{8}{14}\mathit{\text{log}}(\frac{8}{14})+\frac{6}{14}\mathit{\text{log}}(\frac{6}{14})=0.116121$; $H(G_c)=\frac{7}{14}\mathit{\text{log}}(\frac{7}{14})+\frac{7}{14}\mathit{\text{log}}(\frac{7}{14})=-0.693147$; H(φ) = 0. Thus KWII(G_r, G_c) for the marginal table is −H(G_r, G_c) + H(G_r) + H(G_c) = 0.799029. By the Lemma ${\mathit{\text{KWII}}}_P(G_r,G_c)=\frac{1}{2}KWII(G_r,G_c|P=0)+\frac{1}{2}KWII(G_r,G_c|P=1)=-0.0140322$. By the Proposition KWII(G_r, G_c, P) = KWII_P(G_r, G_c) −KWII(G_r, G_c) = −0.0140322 − 0.799029 6 ≠ 0, indicating KWII interaction.

In contrast, odds ratio is 2 for the left table and $\frac{1}{2}$ for the right table. Interaction on the odds ratio scale is indicated by the differing odds ratios. This example highlights the fact that KWII interaction is not a difference in the information across strata, but the change in information as a consequence of stratification.

4.1. Global properties of KWII

In general, factorizations of the density (equivalently, additive decompositions of the log density) determine the dependence relationships among the variables. A simple example is the independence of x and y, where the joint density factors into separate functions of x and y:

$$f(xy)=f(x)f(y)$$

Another example is no interaction of the x, y association with z:

$$f(xyz)=f(xy|z)f(z)=f(xy)f(z)$$

so the x, y association is constant over levels of z. Here the absence of the 3-way interaction is reflected by the fact that no factor is a function of more than two variables.

4.1.1. KWII and factorization of joint density: example

A dependence model is of the form

$$f(x)=\prod _A{g}_A(x_A)$$ (6)

where A ranges over some subsets of indices, and g is a general function (i.e., not necessarily a density function). We assume that each g cannot be further factored. Such a factorization completely determines the dependencies among the variables.

We will show that a particular factorization of the density function leads to the KWII. Considering the case of three variables x; y; z demonstrates the general idea. Assuming strictly positive densities can write the identity:

$$\begin{array}{lll}f(x,y,z)& =& \underset{}{\underbrace{f(x)}}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\underset{}{\underbrace{f(y)}}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\underset{}{\underbrace{f(z)}}\underset{}{\underbrace{\frac{f\hspace{0.17em}(x,y)}{f\hspace{0.17em}(x)f\hspace{0.17em}(y)}}}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\underset{}{\underbrace{\frac{f\hspace{0.17em}(x,z)}{f\hspace{0.17em}(x)f\hspace{0.17em}(z)}}}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\underset{}{\underbrace{\frac{f\hspace{0.17em}(y,z)}{f\hspace{0.17em}(y)f\hspace{0.17em}(z)}}}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\underset{}{\underbrace{\frac{f\hspace{0.17em}(x,y,z)f\hspace{0.17em}(x)f\hspace{0.17em}(y)f\hspace{0.17em}(z)}{f\hspace{0.17em}(x,y)f\hspace{0.17em}(x,z)f\hspace{0.17em}(y,z)}}}\\ & =& g_x(x)\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{g}_y(y)\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{g}_z(z)\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{g}_{x,y}(x,y)\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{g}_{x,z}(x,z)\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{g}_{y,z}(y,z)\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{g}_{x,y,z}(x,y,z)\end{array}$$ (7)

Then:

$$\begin{array}{lll}\mathit{\text{log}}\hspace{0.17em}f(x,y,z)& =& \mathit{\text{log}}(g_x(x))+\mathit{\text{log}}(g_y(y))+\mathit{\text{log}}(g_z(z))+\mathit{\text{log}}(g_{x,y}(x,y))+\mathit{\text{log}}(g_{x,z}(x,z))\\ & & +\mathit{\text{log}}(g_{y,z}(y,z))+\mathit{\text{log}}(g_{x,y,z}(x,y,z))\end{array}$$ (8)

From (7), (8), and the definition of entropy and KWII we see that:

$$\begin{array}{lll}E[\mathit{\text{log}}\hspace{0.17em}f(x,y,z)]& =& \mathit{\text{KWII}}(x)+\mathit{\text{KWII}}(y)+\mathit{\text{KWII}}(z)+\mathit{\text{KWII}}(x,y)\\ & & +\mathit{\text{KWII}}(x,z)+\mathit{\text{KWII}}(y,z)+\mathit{\text{KWII}}(x,y,z).\end{array}$$ (9)

For example, referring to (7):

$$\begin{array}{lll}E[\text{log}\hspace{0.17em}{g}_{x,\hspace{0.17em}y,\hspace{0.17em}z}(x,\hspace{0.17em}y,\hspace{0.17em}z)]\hfill & =\hfill & E[\text{log}\left(\frac{f(x,\hspace{0.17em}y,\hspace{0.17em}z)\hspace{0.17em}f(x)\hspace{0.17em}f(y)\hspace{0.17em}f(z)}{f(x,\hspace{0.17em}y)\hspace{0.17em}f(x,\hspace{0.17em}z)\hspace{0.17em}f(y,\hspace{0.17em}z)}\right)\hfill \\ \hfill & =\hfill & -H(x,\hspace{0.17em}y,\hspace{0.17em}z)+H(x,\hspace{0.17em}y)+H(x,\hspace{0.17em}z)+H(y,\hspace{0.17em}z)-H(x)-H(y)-H(z)\hfill \\ \hfill & =\hfill & \mathit{\text{KWII}}(x,\hspace{0.17em}y,\hspace{0.17em}z)\hfill \end{array}$$

In this way, we can pair each factor in (7) with a corresponding KWII. KWII(x_A) corresponds to the expectation of log g_A(x_A), which will be zero when g_A(x_A) ≡ 1, i.e., there is no factor involving all the indices in A and thus no interaction of that order. Otherwise, all variables in A must be considered simultaneously to describe the probability distribution and no simplification is possible. KWII(X_A) = 0 when the average value of g_A(X_A) is 1, and measures interaction using the expected log likelihood.

4.1.2. KWII and factorization of joint density: general result

The derivation of the general result corresponding to (9) relies on the Möbius transform defined in section 2.5. For positive densities define

$$\Gamma (z)=\sum _{w\subseteq z}{(-1)}^{|z-w|}\text{log}\hspace{0.17em}f(w)$$ (10)

so that:

$$E[\Gamma (z)]=\sum _{w\subseteq z}{(-1)}^{|z-w|}E[\text{log}\hspace{0.17em}f(w)]=-\sum _{w\subseteq z}{(-1)}^{|z-w|}H(w)=\mathit{\text{KWII}}(z)$$

We note that (10) is a Möbius transform, thus the inverse transform gives log f (x) = Σ_z⊆x Γ(z) so the desired factorization of the joint density is f (x) = Π_z⊆x e^Γ(z). Finally,

$$-H(X)=E[\text{log}\hspace{0.17em}f(x)]=\sum _{z\subseteq x}E[\Gamma (z)]=\sum _{z\subseteq x}\mathit{\text{KWII}}(z).$$ (11)

The above equation can be viewed as substituting the factorization of e^−H(X) for the factorization of the density function.

4.2. Dependency structure

The sum of all non-singleton KWII accounts for the total correlation information since by (3) and (11) TCI(X) = Σ_{Z⊆X,|Z|≥2}KWII(Z) and the KWII of a singleton is the negative entropy. Sets of null KWII reflect global dependency statements. For example, note that if g_x,y,z(x, y, z) ≡ 1 and g_x,z(x, z) ≡ 1 in (7), then the density has a factorization h(x, y)h(y, z) so that X and Z are conditionally independent given Y. This implies KWII(X, Y, Z) = KWII(X, Y) = 0. Likewise, if Y and Z are conditionally independent then KWII(X, Y, Z) = KWII(Y, Z) = 0. When the three variables are independent KWII(X, Y, Z) = KWII(X, Y) = KWII(Y, Z) = KWII(X, Z) = 0.

4.3. Relation to log-linear statistical models

A decomposition of the form (6) but different from (7) is the log-linear model for contingency tables, e.g.,

$$\text{log}\hspace{0.17em}f(x)=u_1(x_1)+u_2(x_2)+u_3(x_3)+u_{1,2}(x_1,x_2)+\dots$$

The interpretation of the log-linear model relies on the implied factorization of the density in the same way as (7). To gain insight into KWII we will compare it with log-linear modeling in three examples.

Example 4.1: Two binary variables

The joint and marginal probabilities are presented in the following table.

$$\begin{array}{ccc}{p}_{11}& p_{10}& p_{1·}\\ p_{01}& p_{00}& p_{0·}\\ p_{·1}& p_{·0}& \end{array}$$

The log-linear model factors the density such that for (x₁, x₂) in {0,1}²:

$$\text{log}\hspace{0.17em}f(x_1,x_2)={\mu }_0+{\mu }_1{x}_1+{\mu }_2{x}_2+{\mu }_{12}{x}_1{x}_2$$ (12)

where μ₀ = log p₀₀, ${\mu }_1=\text{log}\frac{p_{10}}{p_{00}}$, ${\mu }_2=\text{log}\frac{p_{01}}{p_{00}}$ and ${\mu }_{12}=\text{log}\frac{p_{11}{p}_{00}}{p_{01}{p}_{10}}$. μ₁₂ is the log cross-product ratio, where the cross-product ratio of a 2x2 table is defined as $cpr=\frac{p_{11}{p}_{00}}{p_{10}{p}_{01}}$. The parameter μ₁₂ is a measure of the association x₁ and x₂, which is KWII interaction for the simple case of the 2x2 table.

With information metrics, to get a factorization like (7) we write:

$$\begin{array}{l}\text{log}\hspace{0.17em}f(x_1,x_2)=\text{log}\hspace{0.17em}f(x_1)+\text{log}\hspace{0.17em}f(x_2)+\text{log}\frac{f(x_1,x_2)}{f(x_1)f(x_2)}\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}=\text{log}{p}_{1·}^{x_1}{p}_{0·}^{1-x_1}+\text{log}{p}_{·1}^{x_2}{p}_{·0}^{1-x_2}+\text{log}\frac{f(x_1,x_2)}{f(x_1)f(x_2)}\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}=x_1\text{log}{p}_{1·}+(1-x_1)\text{log}{p}_{0·}+x_2\text{log}{p}_{·1}+(1-x_2)\text{log}{p}_{·0}+\text{log}\frac{f(x_1,x_2)}{f(x_1)f(x_2)},\end{array}$$

so that

$$E[\text{log}\hspace{0.17em}f(x_1,x_2)]=-H(x_1)-H(x_2)+\mathit{\text{Inf}}(x_1\perp x_2)$$

KWII interaction (i.e. association/correlation in this case) is measured by Inf (x₁ ⊥ x₂), which equals 0 iff x₁ and x₂ are independent. An alternate description of Inf (x₁ ⊥ x₂) = 0 which better generalizes to interaction among more than two variables is H(X₁) = H_X2(X₁) by (2).

It is informative to inspect the last term above, $\mathit{\text{Inf}}(x_1\perp x_2)=KWII(x_1,x_2)=E[\text{log}\frac{f(x_1,x_2)}{f(x_1)f(x_2)}]$ in detail:

$$E[\text{log}\frac{f(x_1,x_2)}{f(x_1)f(x_2)}]=p_{11}\text{log}\frac{p_{11}}{p_{1·}{p}_{·1}}+p_{10}\text{log}\frac{p_{10}}{p_{1·}{p}_{·0}}+p_{00}\text{log}\frac{p_{00}}{p_{0·}{p}_{·0}}+p_{01}\text{log}\frac{p_{01}}{p_{0·}{p}_{·1}}$$

Each term corresponds to a cell of the 2 x 2 table going clockwise from the upper left, and is a measure of independence in that cell weighted by probability of that cell. The sum is thus a weighted average of independence metrics. This is reminiscent of the chi-squared test for independence in a 2 x 2 table, which combines within-cell measures of deviation from independence.

To compare the log-linear interaction parameter μ₁₂ and KWII(x₁, x₂) let us assume that the log-linear model holds so that (12) implies that KWII(x₁, x₂) is the expectation of

$$\begin{array}{l}\text{log}\frac{f(x_1,x_2)}{f(x_1)f(x_2)}=\text{log}\hspace{0.17em}f(x_1,x_2)-\text{log}\hspace{0.17em}f(x_1)-\text{log}\hspace{0.17em}f(x_2)\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}={\mu }_0+{\mu }_1{x}_1+{\mu }_1{x}_2+{\mu }_{12}{x}_1{x}_2\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}-x_1\text{log}\hspace{0.17em}{p}_{1·}-(1-x_1)\text{log}\hspace{0.17em}{p}_{0·}-x_2\text{log}\hspace{0.17em}{p}_{·1}-(1-x_2)\text{log}\hspace{0.17em}{p}_{·0}\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}={\mu }_0-\text{log}\hspace{0.17em}{p}_{0·}-\text{log}\hspace{0.17em}{p}_{·0}+x_1({\mu }_1-\text{log}\hspace{0.17em}{p}_{1·}+\text{log}\hspace{0.17em}{p}_0.)\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}+x_2({\mu }_2-\text{log}\hspace{0.17em}{p}_{·1}+\text{log}\hspace{0.17em}{p}_{·0})+x_1{x}_2{\mu }_{12}\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}=x_1{x}_2{\mu }_{12}+\text{log}\frac{p_{00}}{p_{0·}{p}_{·0}}+x_1\text{log}\frac{p_{10}/p_{00}}{p_{1·}/p_{0·}}+x_2\text{log}\frac{p_{01}/p_{00}}{p_{·1}/p_{·0}}\end{array}$$

We see that KWII includes the log-linear parameter ${\mu }_{12}=\text{log}\frac{p_{11}{p}_{00}}{p_{01}{p}_{10}}$ as a component, plus three additional components which seem to also reflect the notion of interaction. For example, nonzero $\frac{p_{00}}{p_{0·}{p}_{·0}}$ is a deviation from the statistical independence of x₁ and x₂, while $\frac{p_{10}/p_{00}}{p_{1·}/p_{0·}}$ contrasts the odds of x₁ when x₂ = 0 to the odds of x₁ ignoring x₂ to measure the heterogeneity of the x₁ effect with respect to x₂. A similar interpretation holds for $\frac{p_{01}/p_{00}}{p_{·1}/p_{·0}}$.

When row and column sums are fixed, μ₁₂ is unaffected by marginal probabilities and is known to capture all of the conditional information about the x₁, x₂ association, ie. it is the sufficient statistic with respect to the conditional distribution. The KWII is affected by the marginal probabilities and is not conditional on the margins. This is further discussed in section 4.7.

Example 4.2: Three binary variables

The log-linear model factors the density such that:

$$\text{log}\hspace{0.17em}f(x_1,x_2,x_3)={\mu }_0+{\mu }_1{x}_1+{\mu }_2{x}_2+{\mu }_3{x}_3+{\mu }_{12}{x}_1{x}_2+{\mu }_{13}{x}_1{x}_3+{\mu }_{23}{x}_2{x}_3+{\mu }_{123}{x}_1{x}_2{x}_3$$

The 2-way -terms are calculated from 2 x 2 marginal tables as in the previous example. The 3-way -term is log $\frac{\mathit{\text{cpr}}(x_1,x_2|x_3=1)}{\mathit{\text{cpr}}(x_1,x_2|x_3=0)}$, where cpr is the cross product ratio of the corresponding marginal 2x2 table. It measures the effect of x₃ on the x₁, x₂ association.

With information metrics we write the expected log density as:

$$E\left[\text{log}\left(f(x_1)f(x_2)f(x_3)\frac{f(x_1,x_2)}{f(x_1)f(x_2)}\frac{f(x_1,x_3)}{f(x_1)f(x_3)}\frac{f(x_2,x_3)}{f(x_2)f(x_3)}\frac{f(x_1,x_2,x_3)f(x_1)f(x_2)f(x_3)}{f(x_1,x_2)f(x_1,x_3)f(x_2,x_3)}\right)\right]$$

The 3-way interaction derives from the last factor:

$$\begin{array}{l}\mathit{\text{KWII}}(x_1,x_2,x_3)=E\left[\text{log}\frac{f(x_1,x_2,x_3)f(x_1)f(x_2)f(x_3)}{f(x_1,x_2)f(x_1,x_3)f(x_2,x_3)}\right]\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}=E\left[\text{log}\left(\frac{f(x_1,x_2,x_3)f(x_3)}{f(x_1,x_3)f(x_2,x_3)}\right)\left(\frac{f(x_1)f(x_2)}{f(x_1,x_2)}\right)\right]\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}=\mathit{\text{Inf}}(x_1\perp x_2|x_3)-\mathit{\text{Inf}}(x_1\perp x_2)\end{array}$$

Similarly to the log-linear model the 3-way term represents the effect of effect of x₃ on the x₁, x₂ association.

Example 4.3: Two trinomial genotype variables

Here x₁ and x₂ take values in {0, 1, 2} representing genotype. In this case the log-linear model uses four parameters to model the x₁, x₂ interaction, μ₁₂(1, 1), μ₁₂(1, 2), μ₁₂(2, 1), μ₁₂(2, 2). In contrast, only the single metric KWII(x₁, x₂) is required. This reduction of degrees of freedom is different from that resulting from recoding genotypes to model restricted genetic models such as dominant or recessive, as it makes no genetic assumptions and relies only on the distribution of genotypes. In general, for a kth-order interaction of trinomial genotype variables the log-linear model is expressed by 2^k parameters, as opposed to a single KWII. So the information-based model is more parsimonious, especially for high order interactions. There is a loss of detail, but since the number of parameters poses a huge challenge when screening genome-wide data for interactions the principled reduction of parameters inherent in the information approach is advantageous. For example, if we are analyzing haplotypes with r variants, a k^th order interaction is expressed by (r − 1)^k interaction parameters using the log-linear parameterization, but only a single KWII.

4.4. Relation of KWII and PAI

By (4) and (11),

$$\begin{array}{lll}\mathit{\text{PAI}}(X,P)& =& H(X)+H(P)-H(X,P)\\ & =& -{\sum }_{Z\subseteq X}\mathit{\text{KWII}}(Z)+H(P)+{\sum }_{Z\subseteq X\cup P}\mathit{\text{KWII}}(Z)\\ & =& {\sum }_{Z\subseteq X}\mathit{\text{KWII}}(Z,P)+H(P)\end{array}$$ (13)

PAI is an imperfect but useful tool for screening potential interactions. In the AMBIENCE algorithm to be described later an interaction term KWII(Z) is considered whenever PAI(Z) is large. However a KWII can be either positive or negative, so non-zero terms in (13) could conceivably cancel each other to make PAI small when there are interactions present. However Han (1980) shows that KWII is nonnegative when the distribution is sufficiently close to complete independence so cancelation may be considered unlikely, supporting AMBIENCE’s use of PAI as a screening tool for interactions.

There is a converse to (13). Since by (13) PAI(X, P) −H(P)=Σ_Z⊆X KWII(Z, P), Möbius inversion gives

$$\begin{array}{l}\mathit{\text{KWII}}(X,P)={\sum }_{Z\subseteq X}{(-1)}^{|X-Z|}(PAI(Z,P)-H(P))\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}={\sum }_{Z\subseteq X}{(-1)}^{|X-Z|}PAI(Z,P)\end{array}$$

since Σ_Z⊆X (−1)^|X−Z|H(P) = H(P)Σ_Z⊆X (−1)^|X−Z|(−1)^|X−Z| = 0.

4.5. Similarity to components of variance

We can write a decomposition of the entropy of the phenotype P. By (4) and (13)

$$H(P)=\sum _{\phi \subset z\subseteq x}\mathit{\text{KWII}}(Z,P)+H_X(P)$$

Since the last term above is a residual, the expression is of a form similar to components of variance, with the distinction that these components can be positive or negative.

4.6. Statistical distribution of KWII

As was the case for PAI, for multinomial data the sample estimate of KWII is obtained by substituting maximum likelihood estimates of the cell probabilities into the definition. Han (1980) shows that if a condition called semi-independence holds then 2N KWII(X) converges in distribution asymptotically to a chi-squared statistic with degrees of freedom Π(r_Xi − 1), where r_A is the number of values the random variable A can take. Further, if X ≠ Y, 2N KWII(X) and 2N KWII(Y) asymptotically approach independent chi-squared distributions.

Han’s results are based on a Taylor series expansion around the distribution for which the variables are completely independent, and so are accurate for cases close to that condition. Since we wish to employ our methods when strong correlations exist the practical usefulness of the asymptotic results needs to be established. Also, the asymptotic relation of semi-independence conditions to the factorization of the density needs to be investigated. Further study is required to justify the use of these results in applications.

4.7. Related information approaches

Clayton (2009) gives a stimulating critique of KWII, which he terms “synergy”. He advocates an alternative information metric for interaction based on the uncertainty remaining if all lower order relationships were known. Good(1963) derived the associated metric to be the maximum entropy over all possible distributions constrained by fixing the lower order margins, and showed that this corresponds to interaction in a logistic regression model. Our discussion of example 4.1 supports Clayton’s observation that KWII in influenced by the marginal probabilities. In the simple two variable case the argument for the specification of fixed margins in the analysis of 2 x 2 tables is not conclusive. In that case the odds ratio with fixed margins (a logistic model parameter) and unconditional inference (e.g. via a chi-squared test) are both acceptable. This suggests the comparison of KWII and Good’s maximum entropy may be analogous to the contrast between conditional and unconditional approaches to statistical inference for contingency tables. Agresti (2002) discusses the relative merits of those two approaches. To summarize, there is information about association in the table margins (Agresti, 2002; Zhu and Reid, 1994), but it is a small amount. Future work will consider this issue for genetic applications and general dimensionality. Conditional inference methods for multinomial data tend to be computationally complex, and this may be an important consideration.

Clayton also shows the value of KWII is not invariant under case-control sampling. The seriousness of this concern will be addressed in future work. The important question is whether inferences are rendered invalid. KWII is not really an interpretable parameter in the way that odds ratio contrasts are; it is used screening rather than for interpreting the strength of effects. So this criticism, while valid for the magnitude of the metric, may not discourage the use of KWII for case-control studies. This is an important question which needs further study.

Reconstructability Analysis (RA) (Zwick, 1994; Shervais et al., 2010) is an information-based approach related to maximum entropy. This approach reduces the dimension from the saturated model by projecting the joint distribution onto a set of margins. The margins specify the reduced model. The marginal distributions are then composed to form a more parsimonious estimate of the joint distribution whose fit to the data will be tested using Good’s maximum entropy method to evaluate the model. Thus the comments of the previous paragraph also apply to RA.

Maximum entropy and RA are closely related to log-linear and logistic modeling. Importantly, they require iterative computational methods, unlike KWII. Thus KWII may be the only feasible choice for high-dimensional model search.

5. Model search

The non-iterative computation required by the information metrics of this paper makes them candidates for the large-scale screening of genetic interactions, when many other methods (e.g. logistic regression, reconstructability analysis) are infeasible. However, large problems are still daunting. Using the definition to compute KWII for all possible interactions (i.e. all possible combinations of variables) is of complexity O(3^N), where N is the number of variables. Thus efficient algorithms and heuristics must be used. The AMBIENCE algorithm is a heuristic search strategy for interaction detection.

5.1. The AMBIENCE algorithm

The AMBIENCE algorithm (Chanda et al., 2008) screens to find sets of variables with significant PAI of order ≤ k, where the significance is determined by the chi-squared test or the permutation method given in section 3.1. It does this greedily, building candidate i^th-order PAI by adding a new variable to a significant (i − 1)^th-order PAI. Finally for each PAI selected the KWII with the same arguments is tested. Note that a significant PAI could be due to any of the KWII given by (13), not just the highest order one. AMBIENCE accounts for this by its greedy choice of PAI, which nests the interaction orders. A i^th-order PAI is not tested unless it contains an (i − 1)^th-order PAI, which contains a significant (i − 2)^th-order PAI, and so on. For example, this means that a two-way interaction between SNPs is not tested unless there is significant marginal association between at least one of the SNPs and the disease phenotype. Testing the associated nested KWII can be viewed as a heuristic approximation to testing all the KWII implied by (13) for a significant PAI.

5.2. The Möby Quick algorithm

Thoma (1991) gave an efficient algorithm, the Fast Möbius Transform (FMT), for computing KWII(X) for all subsets X. As presented, the FMT calculates all KWII up to and including order N in O(N2^N) time, only a factor of N greater than the Nth order interaction alone. As it stands, the FMT is not of use to us since it is infeasible to consider very high order interactions orders and we limit the order to k << N. Happily, the basic FMT is easily modified to efficiently calculate KWII only up to order k. The modified algorithm, of complexity O(kN^k+1), is given in the Appendix.

We will usually wish to restrict the interactions so that every one includes the same fixed size c set of variables, For example, in our application every interaction of interest involves the phenotype P, so c = 1. We may also wish to fix other variables, e.g., gender, so that every interaction involves both phenotype and gender so c = 2. The new algorithm enables this to be done, and the complexity is reduced to O(kcN^k + k(N − c)^k+1). Although the complexity is polynomial in N of order k + 1, N can be huge. It will be necessary to reduce N, possibly by selecting those variables found to play a role in significant PAI.

In this paper we propose an algorithm, Möby Quick, which is complementary to AMBIENCE, based on the modification of the FMT given in the Appendix. After significant PAI are identified by AMBIENCE it is usually the case that a small subset Θ of variables are seen to be participating in the interactions. Rather than computing KWII individually as in AMBIENCE, Möby Quick computes all possible interactions up to order k involving variables in Θ in O(k |Θ|^k+1) time. The algorithm casts a wider net than AMBIENCE, and if |Θ| is small it is computationally feasible.

6. Discussion

In this paper we presented a likelihood factorization which identifies quantities which capture the notion of statistical interaction. The information theoretic interaction metrics arise from the expectation of the log-likelihood and sample estimates of all the metrics we have considered arise from maximum likelihood estimates of the entropies. An advantage of the information approach is that the meaning of the metrics and their connection to the statistical distribution of the data is clear, as opposed to many machine learning algorithms for interaction detection.

For categorical variables with more than two levels the expected log-likelihood reduces the number of parameters compared to log-linear models and logistic regression. This gives a much more parsimonious description of the interactions among the variables. Some detail is lost, but parsimony may be a more important consideration when searching for interactions in high dimensional data. In addition, the computations required are non-iterative, which greatly extends their feasibility compared with methods requiring numerical optimization such as logistic regression and reconstructability analysis.

The calculus of the information metrics facilitates a variety of re-expressions of the metrics, which suggest different approaches to model search and computation. The potential algorithms suggested in this paper will be explored in future work.

The goal of this paper was to connect information-theoretic metrics to probability models so that they can be better understood in the context of data analysis. Along the way we also uncovered some differences. For genetic markers with many levels there are many fewer parameters produced in the information approach. This may be critical in the case of haplotype analysis, where for haplotypes with r variants, a k^th order interaction is expressed by (r − 1)^k interaction parameters using the log-linear parameterization, but only a single KWII. The way the metrics are defined also makes transformations among them easy using powerful algorithmic tools.

We have also highlighted areas in need of further development. The distribution of KWII needs further study. Clayton (2009) has shown that the influence of the sampling plan and the influence of marginal probabilities will be key question in future comparisons of interaction detection methods. Also, we will explore improved modeling strategies using these metrics. In most of the algorithms quantities are not refined based on previous modeling steps. That is, a KWII at the end of AMBIENCE is the same number as at the beginning. The AMBIENCE algorithm just tells you that you should look at it. Our future work will explore approaches where previous decisions do simplify the computation of KWII, essentially using fewer parameters.

APPENDIX: Modified FMT

The following algorithm calculates KWII for all variable subsets of orders = k, each including all of the variables in a fixed set C. Let Ω be all size = k subsets of the complete set of N variables. For a fixed set C of the variables let Ω⁻ represent all the variable sets in Ω which contain the variables in C. Let initially

$$f_0(S)=H(S)$$

for all S ∈ Ω. Iterate for j ranging over the variables in C for S ∈ Ω as follows:

$$f_j(S)=\{\begin{array}{ll}{f}_{j-1}(S)\hfill & \text{if}\hspace{0.17em}j\notin S\hfill \\ -f_{j-1}(S-\{j\})+f_{j-1}(S)\hfill & \text{if}\hspace{0.17em}j\in S\hfill \end{array}$$ (14)

Then iterate 14 for j ranging over the variables in C̄ for S ∈ Ω⁻. Finally, after step N when all variables have been iterated over, set KWII(S) = f_N(S), S ∈ Ω⁻.

Proof: If k = N and C = φ, the algorithm is the original FMT. If k < N and C = φ, a iteration 14 for a set S depends only on subsets of S and thus is unaffected. If C ≠ φ, the initial iteration over C is the same as the basic FMT. For the second stage iterating over Ω⁻, a step of (14) for a set S containing C depends only on lower-order subsets containing C, which have been computed.