R²-equitability is satisfiable

Kinney and Atwal (1) make excellent points about mutual information, the maximal information coefficient (2, 3), and “equitability.” One of their central claims, however, is that, “No nontrivial dependence measure can satisfy $R^2$-equitability.” We argue that this is the result of a poorly constructed definition, which we quote:

“A dependence measure $D\left[X;Y\right]$ is $R^2$-equitable if and only if, when evaluated on a joint probability distribution $p\left(X,Y\right)$ that corresponds to a noisy functional relationship between two real random variables X and Y, the following relation holds:

$$D\left[X;Y\right]=g\left(R^2\left[f\left(X\right);Y\right]\right).$$

Here, g is a function that does not depend on $p\left(X,Y\right)$ and f is the function defining the noisy functional relationship, i.e.,

$$Y=f\left(X\right)+\eta ,$$

for some random variable η. The noise term η may depend on $f\left(X\right)$ as long as η has no additional dependence on X….”

This definition is undone by the unconventional specification of the noise term. Specifically, allowing η to depend arbitrarily on $f\left(X\right)$ lets many different combinations of f and η result in the same $p\left(X,Y\right)$. For example, consider $f_1\left(X\right)=X^2$ and ${\eta }_1=\mathcal{N}\left(0,1\right)$, against $f_2\left(X\right)=X$ and ${\eta }_2=-f_2\left(X\right)+f_2{\left(X\right)}^2+\mathcal{N}\left(0,1\right)$. The resulting $p\left(X,Y\right)$ distributions are identical, but $R^2\left[f_1\left(X\right);Y\right]\ne R^2\left[f_2\left(X\right);Y\right]$—a consequence of the deterministic trend embedded in ${\eta }_2$.

We emphasize the cause of the definitional deficiency (which the authors exploit to demonstrate unsatisfiability) because it suggests an immediate fix: make η trendless. By constraining the expectation $E\left[\eta |f\left(X\right)\right]=0$, the identifiability issue is resolved without limiting expressive power: any trend removed from η can, and should, be included in $f\left(X\right)$ instead. Under this formulation, we also see no reason to restrict the dependence of η to $f\left(X\right)$ alone; it can depend arbitrarily on X, as long as $E\left[\eta |X\right]=0$.

Without a trend in η, not only does the resulting definition of $R^2$-equitability escape Kinney and Atwal’s reductio, but it is demonstrably satisfiable. Because $E\left[\eta |X\right]=0\Rightarrow f\left(X\right)=E\left[Y|X\right]$, $R^2\left[f\left(X\right);Y\right]$ is determined by $p\left(X,Y\right)$, satisfying the modified definition with g as the identity function. Further, in the large sample limit (for nonpathological functions), $\widehat{f}\left(X\right)\approx f\left(X\right)$ is estimable from X, Y, yielding increasingly accurate approximations of $R^2\left[\widehat{f}\left(X\right);Y\right]\approx R^2\left[f\left(X\right);Y\right]$, suggesting a family of schemes for nonparametric estimation of $D\left[X;Y\right]$ that satisfy $R^2$-equitability.

$R^2$-equitable measures of dependence care only about how accurately Y can be predicted—under a quadratic loss function—by X and are thus sensitive to nonlinear transformations of Y and not symmetric $\left(D\left[X;Y\right]\ne D\left[Y;X\right]\right)$, in contrast to any dependence measure satisfying Kinney and Atwal’s self-equitability (1). These two distinct notions of equitability are useful in different circumstances: $R^2$-equitability should be preferred when quantifying how well you can predict an outcome in expectation (measuring your least-squares predictive accuracy), and measures satisfying self-equitability (exemplified by mutual information) may be more appropriate when quantifying how well you can predict Y in probability, being sensitive to how the distribution $p\left(Y|X\right)$ varies with X.

Thus, a simple modification of Kinney and Atwal’s definition renders a satisfiable notion of $R^2$-equitability that is usefully distinct from the notion of self-equitability the authors propose (1). Both can coexist.