On the Versatile Uses of Partial Distance Correlation in Deep Learning

Abstract

Comparing the functional behavior of neural network models, whether it is a single network over time or two (or more networks) during or post-training, is an essential step in understanding what they are learning (and what they are not), and for identifying strategies for regularization or efficiency improvements. Despite recent progress, e.g., comparing vision transformers to CNNs, systematic comparison of function, especially across different networks, remains difficult and is often carried out layer by layer. Approaches such as canonical correlation analysis (CCA) are applicable in principle, but have been sparingly used so far. In this paper, we revisit a (less widely known) from statistics, called distance correlation (and its partial variant), designed to evaluate correlation between feature spaces of different dimensions. We describe the steps necessary to carry out its deployment for large scale models – this opens the door to a surprising array of applications ranging from conditioning one deep model w.r.t. another, learning disentangled representations as well as optimizing diverse models that would directly be more robust to adversarial attacks. Our experiments suggest a versatile regularizer (or constraint) with many advantages, which avoids some of the common difficulties one faces in such analyses ¹.

1. Introduction

The extent to which popular architectures in computer vision even partly mimic human vision continues to be studied (and debated) in our community. But consider the following hypothetical scenario. Let us say that a fully functional computational model of the visual system – perhaps a modern version of the Neocognitron [20] – was somehow provided to us. And we wished to “compare” its behavior to modern CNN models [33,28]. To do so, two options appear sensible. The first – inspired by analogies between computational vision and biological vision – would draw a correspondence between how simple/complex cells in the visual cortex process scenes and their induced receptive fields with those of activations of units/blocks in a modern deep neural network architecture [60]. While this process is often difficult to carry out systematically, it is powerful and, in some ways, has contributed to interest in biologically inspired deep learning, see [67]. Updated forms of this intuition – associating different subsets of cells (or neural network units) to different semantic/visual concepts – remains the default approach we use in debugging and interpretation. The second option for tackling the hypothetical setting above is to pose it in an information theoretic setting. That is, for two models ${\Theta }_X$ and ${\Theta }_Y$, we ask the following question: what has ${\Theta }_X$ learned that ${\Theta }_Y$ has not? Or vice versa. The asymmetry is intentional because if we consider two random variables (r.v.) $X,Y$, the question simply takes the form of “conditioning”, i.e., compare $\mathbb{P}\left(X\right)$ versus $\mathbb{P}\left(X\mid Y\right)$. This form suffices if our interest is restricted to the predictions of the two models. If we instead wish to capture the model's behavior more globally – when $X$ and $Y$ denote the full set of feature responses – we can use divergence measures on high dimensional probability measures given by the two models (${\Theta }_X$ and ${\Theta }_Y$) responses on the training samples. Importantly, notice that our description assumes that, at least, the probability measures are defined on the same domain.

More general use cases.

While the above discussion was cast as comparing two networks, it is representative of a broad basket of tasks in deep learning. (a) Consider the problem of learning fair representations [71,17,70,44] where the model must be invariant to one (or more) sensitive attributes. We seek latent representations, say ${\Psi }_{\text{pred}}\left(X\right)$ for the prediction task, which minimizes mutual information w.r.t. the latent representation relevant for predicting the sensitive attribute ${\Psi }_{\text{sens}}\left(X\right)$. Indeed, if information regarding the sensitive attribute is partially preserved or leaks into ${\Psi }_{\text{pred}}\left(X\right)$, the relative entropy will be low [49]. Observe that this calculation is possible partly because the latent space specifies the same probability space for the two distributions. (b) The setting is identical in common approaches for learning disentangled representations, where disentanglement is measured via various information theoretic measures [8,1,21,61]. If we now segue back to comparing two different networks, but without the convenience of a common coordinate system to measure divergence, the options turn out to be limited. (c) Recently, in trying to understand whether vision Transformers “see” similar to convolutional neural networks [56], one option utilized recently was a kernel-based representation similarity, in a layer-by-layer manner. What we may actually want is a mechanism for conditioning – for example, if one of the models is thought of a “nuisance variable”, we wish to check the residual in the other after the first has been controlled for (or marginalized out). Importantly, this should be possible without assuming that the probability distributions live in the same space (or networks ${\Theta }_X$ and ${\Theta }_Y$ are the same).

A direct application of CCA?

Consider two different feature spaces ($\mathcal{X}$ and $\mathcal{Y}$), say in dimensions ${ℝ}^p$ and ${ℝ}^q$, pertaining to feature activations from two different models. Comparison of these two feature spaces is possible. One natural choice is canonical correlation analysis (CCA) [5], a generalization of correlation, specifically suited when $p\ne q$. The idea has been utilized for studying representation similarity in deep neural network models [48], albeit in a posttraining setting for reasons that will be clear shortly, as well as for identifying more efficient training regimes (i.e., can lower layers be sequentially frozen after a certain number of timesteps). CCA has also been shown to be implementable within DNN pipelines for multi-view training, called DeepCCA [4], although efficiency can be a bottleneck limiting its broader deployment. A stochastic version of CCA suitable for DNN training with mini-batches has been proposed very recently, and strong experimental evidence was presented [47], also see [25]. Given that a stochastic CCA is now available, its extensions to the partial CCA setting are not yet available. If successful, this may eventually provide a scheme, suitable for deep learning, for controlling the influence of one model (or a set of variables) on another model.

This work.

The starting point of this work is a less widely used statistical concept to measure the correlation between two different feature spaces $\left(\mathcal{X},\mathcal{Y}\right)$ of different dimensions, called distance correlation (and the method of dissimilarities). In shallow settings, CCA and distance correlation offers very similar functionality – for the most part, they can be used interchangeably although distance correlation would also need specification of distances (or dissimilarities). In other words, CCA may be easier to deploy. On the other hand, deep variants of CCA involve specialized algorithms [4,47]. Further, deep versions of partial CCA have not been reported. In contrast, as long as feature distances can be calculated, the differences between the shallow and the deep versions of distance correlation are minimal at best, and adjustments needed are quite minor. These advantages carry over to partial distance correlation, directly enabling conditioning one model w.r.t. another (or using such a term as a regularizer). The main contribution of this paper is to study distance correlation (and partial distance correlation) as a powerful measure in a broad suite of tasks in vision. We review the relevant technical steps which enable its instantiation in deep learning settings and show its broad applications ranging from learning disentangled representations to understanding the differences between what two (or more) networks are learning to training “mutually distinct” deep models (akin to earlier works on $M$ best solutions to MAP estimation in graphical models [19,69]) or training $M$ diverse models for foreground-background segmentation as well as other tasks [27].

1.1. Related Works

Four distinct lines of work are related to our development, which we review next. Similarity between networks. Understanding the similarity between different networks is an active topic [38,24,50] also relevant in adversarial models [15,9]. Early attempts to compare neural network representations were approached via linear regression [58], whose applicability to nonlinear models is limited. As noted above, canonical correlation analysis (CCA) [3,31] is a suitable off-the-shelf method for model comparisons. To this end, singular vector CCA (SVCCA) [55], Projection-Weighted CCA [48], DeepCCA [4], and stochastic CCA [23] are all potentially useful. Recently, [37] studied the invariance properties for a good similarity measurement and proposed the centered kernel alignment (CKA). CKA offers invariance to invertible linear transformations, orthogonal transformations, and isotropic scaling. Separately, [51,56] used CKA to study similarities between deep and wide neural networks and also between different network structures.

Information theoretic divergence measures.

Another body of related work pertains to approximately measuring the mutual information [12] to remove this information, mainly in the context of fair representation learning. Here, mutual information (MI) is measured between features and the sensitive attribute [49]. In [63], another information theoretic bound for learning maximally expressive representations subject to the given attributes is presented. In [10], MI between prediction and the sensitive attributes is used to train a fair classifier whereas [2] describes the use of inverse contrastive loss. Group-theoretic approaches have also been described in [11,45]. The work in [41] gives an empirical solution to remove specific visual features from the latent variables using adversarial training.

Repulsion/Diversity.

If we consider the ensemble of neural networks, there are several different strategies to maintain functional diversity between ensemble members – we acknowledge these results here because they are loosely related to one of the use cases we evaluate later. SVGD [14] shows the benefits of choosing the kernel to measure the similarity between ensemble members. In [13], the authors introduce a kernelized repulsive term in the training loss, which endows deep ensembles with Bayesian convergence properties. The so-called quality diversity (QD) is interesting: [53] tries to maximize a given objective function with diversity to a set of pre-defined measure functions [22,57]. When both the objective and measure functions in QD are differentiable, [18] offers an efficient way to explore the latent space of the objective w.r.t. the measure functions.

Distance correlation (DC).

The central idea motivating our work is distance correlation described in [65]. It has been used in the analysis of nonlinear dependence in time-series [72], and feature screening in ultra high-dimensional data analysis tasks [42] and we will review it in detail shortly.

2. Review: Distance (and Partial Distance) Correlation

Given two random variables $X,Y\in ℝ$ (in the same domain), correlation (say, the Pearson correlation) helps measure their association. One can derive meaningful conclusions by statistical testing. As noted in §1 one generalization of correlation to a higher dimension is CCA, which seeks to find projection matrices such that correlation among the projected data is maximized, see [5].

Benefits of Distance Correlation.

In many applications, the notion of distances or dissimilarities appears quite naturally. Motivated by the need for a scheme that can capture both linear and non-linear correlations when provided with such dissimilarity information, in [65], the authors proposed a new measure of dependence between random vectors, called distance correlation. The key benefits of distance correlation are:

1. The distance correlation $\mathcal{R}$ satisfies $0\le \mathcal{R}\le 1$, and $\mathcal{R}=0$ if and only if $X,Y$ are independent.

2. $\mathcal{R}\left(X,Y\right)$ is defined for $X$ and $Y$ in arbitrary dimensions, e.g., $\mathcal{R}\left(X,Y\right)$ is well-defined when $X$ is of dimension $p$ while $Y$ is of dimension $q$ for $p\ne q$.

We focus on empirical distance correlation for $n$ samples drawn from the unknown joint distribution, and review its calculation.

For an observed random sample $\left(x,y\right)=\left\{\left(X_i,Y_i\right):i=1,\cdots ,n\right\}$ from the joint distribution of random vectors $X$ in ${ℝ}^p$ and $Y$ in ${ℝ}^q$, define:

$$\begin{array}{l}{a}_{k,l}=‖X_k-X_l‖,\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{\overline{a}}_{k,\cdot }=\frac{1}{n}\sum _{l=1}^n{a}_{k,l},{\overline{a}}_{\cdot ,l}=\frac{1}{n}{a}_{k,l},\\ {\overline{a}}_{\cdot ,\cdot }=\frac{1}{n^2}\sum _{k,l=1}^n{a}_{k,l},A_{k,l}=a_{k,l}-{\overline{a}}_{k,\cdot }-{\overline{a}}_{\cdot ,l}+{\overline{a}}_{.,.}\end{array}$$ (1)

where $k,l\in \left\{1,\cdots ,n\right\}$. Similarly, we can define $b_{k,l}=‖Y_k-Y_l‖$, and $B_{k,l}=b_{k,l}-{\overline{b}}_{k,\cdot }-{\overline{b}}_{.,l}+{\overline{b}}_{.,.}$, and based on these quantities we have.

Definition 1.

(Distance correlation) [65]. The empirical distance correlation ${\mathcal{R}}_n\left(x,y\right)$ is the square root of

$${\mathcal{R}}_n^2\left(x,y\right)=\left\{\begin{array}{cc}\frac{{\mathcal{V}}_n^2\left(x,y\right)}{\sqrt{{\mathcal{V}}_n^2\left(x,x\right){\mathcal{V}}_n^2\left(y,y\right)}}& ,{\mathcal{V}}_n^2\left(x,x\right){\mathcal{V}}_n^2\left(y,y\right)>0\\ 0& ,{\mathcal{V}}_n^2\left(x,x\right){\mathcal{V}}_n^2\left(y,y\right)=0\end{array}\right.$$ (2)

where the empirical distance covariance (variance) ${\mathcal{V}}_n\left(x,y\right)$, ${\mathcal{V}}_n\left(x,x\right)$ are defined as ${\mathcal{V}}_n^2\left(x,y\right)=\frac{1}{n^2}{\sum }_{k,l=1}^n{A}_{k,l}{B}_{k,l}$, ${\mathcal{V}}_n^2\left(x,x\right)=\frac{1}{n^2}{\sum }_{k,l=1}^n{A}_{k,l}^2$, with $A$ in (1).

Examples. We show a few simple 2D examples to contrast Pearson Correlation and Distance Correlation in Fig. 1. Notice that if the relationship between the two random variables is not linear, Pearson Correlation might be small while Distance Correlation remains meaningful.

Fig. 1:

Examples of Pearson Correlation and Distance Correlation in different settings. (a): $y=0.5x^2+0.75n,n\sim \mathcal{N}\left(0,1\right)$; (b): $y=0.15x^3+0.75n+2.5,n\sim \mathcal{N}\left(0,1\right)$; (c): $\left[\begin{array}{l}x\hfill \\ y\hfill \end{array}\right]\sim \mathcal{N}\left(\left[\begin{array}{c}0\\ 2.5\end{array}\right],\left[\begin{array}{cc}1& 0.75\\ 0.75& 1.25\end{array}\right]\right)$; (d): $\left[\begin{array}{l}x\hfill \\ y\hfill \end{array}\right]\sim \mathcal{N}\left(\left[\begin{array}{c}0\\ 2.5\end{array}\right],\left[\begin{array}{cc}1& 0\\ 0& 1.25\end{array}\right]\right)$

Extensions to conditioning.

Given three random variables $X$, $Y$, and $Z$, we want to measure the correlation between $X$ and $Y$ but “controlling for” $Z$ (thinking of it as a nuisance variable), i.e., we want to estimate $\mathcal{R}\left(X\left|Z,Y\right|Z\right)={\mathcal{R}}^{\text{*}}\left(X,Y;Z\right)$. Such a quantity is key in existing approaches in disentangled learning, deriving invariant representations and understanding what one or more networks are learning after concepts learned by another network have been accounted for. Consider how this task would be accomplished in linear regression We would project $X$ and $Y$ into the space of $Z$, and only use the residuals to measure the correlation. Nonetheless, defining partial distance correlation is more involved – in [64], the authors introduced a new Hilbert space where we can define the projection of distance matrix. To do so, the authors calculate a $\mathcal{U}$-centered matrix $\tilde{A}$ from the distance matrix $\left(a_{k,l}\right)$ so that the inner product of the $\mathcal{U}$-centered matrices will be the distance covariance.

Definition 2.

Let $A=\left(a_{k,l}\right)$ be a symmetric, real valued $n\hspace{0.17em}\hspace{0.17em}\times \hspace{0.17em}\hspace{0.17em}n$ matrix $(n>2)$ with zero diagonal. Define the $\mathcal{U}$-centered matrix $\tilde{A}=\left({\tilde{a}}_{kl}\right)$ as follows.

$$\begin{gathered} {\tilde{a}}_{kl}=\left\{a_{k,l}-\frac{1}{n-2}\sum _{i=1}^n{a}_{i,l}-\frac{1}{n-2}\sum _{j=1}^n{a}_{k,j}+\frac{1}{\left(n-1\right)\left(n-2\right)}\sum _{i,j=1}^n{a}_{i,j},k\ne l\right. \\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}0\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em},k=l \end{gathered}$$ (3)

Further, the inner product between $\tilde{A}$, $\tilde{B}$ is defined as $\left(\tilde{A}\cdot \tilde{B}\right):=\frac{1}{n\left(n-3\right)}{\sum }_{k\ne l}{\tilde{A}}_{k,l}{\tilde{B}}_{k,l}$, and is an unbiased estimator of squared population distance covariance ${\mathcal{V}}^2\left(x,y\right)$.

Before defining partial distance covariance formally, we recall the definition of orthogonal projection on these matrices.

Definition 3.

Let $\tilde{A}$, $\tilde{B}$, $\tilde{C}$ corresponding to samples $x$, $y$, $z$ respectively, and let $P_{z^{\perp }}\left(x\right)=\tilde{A}-\frac{\left(\tilde{A}\cdot \tilde{C}\right)}{\left(\tilde{C}\cdot \tilde{C}\right)}\tilde{C}$, $P_{z^{\perp }}\left(y\right)=\tilde{B}-\frac{\left(\tilde{B}\cdot \tilde{C}\right)}{\left(\tilde{C}\cdot \tilde{C}\right)}\tilde{C}$ denote the orthogonal projection of $\tilde{A}\left(x\right)$ onto ${(\tilde{C}\left(z\right))}^{\perp }$ and the orthogonal projection of $\tilde{B}\left(y\right)$ onto ${(\tilde{C}\left(z\right))}^{\perp }$.

Now, we are ready to define the partial distance covariance and the partial distance correlation.

Definition 4.

Let $\left(x,y,z\right)$ be a random sample observed from the joint distribution of $\left(X,Y,Z\right)$. The sample partial distance covariance is defined by:

$$pdCov\left(x,y;z\right)=\left(P_{z^{\perp }}\left(x\right)\cdot P_{z^{\perp }}\left(y\right)\right)=\frac{1}{n\left(n-3\right)}\sum _{i\ne j}{\left(P_{z^{\perp }}\left(x\right)\right)}_{i,j}{\left(P_{z^{\perp }}\left(y\right)\right)}_{i,j}$$ (4)

And the partial distance correlation is defined as: ${\mathcal{R}}^{*2}\left(x,y;z\right):=\frac{\left(P_{z^{\perp }}\left(x\right)\cdot P_{z^{\perp }}\left(y\right)\right)}{‖P_{z^{\perp }}\left(x\right)‖‖P_{z^{\perp }}\left(y\right)‖}$ where $‖P_{z^{\perp }}\left(x\right)‖={\left(P_{z^{\perp }}\left(x\right)\cdot P_{z^{\perp }}\left(x\right)\right)}^{1/2}$ is the norm.

Partial distance correlation enables asking various interesting questions. By projecting the original $\mathcal{U}$-centered matrix $\tilde{A}$ onto $\tilde{C}$, the correlation between the residual and $\tilde{B}$ will be a measure of what does $X$ learn that $Z$ does not.

3. Optimizing Distance Correlation in Neural Networks

While distance correlation can be implemented in a differentiable way, and thereby used as an appropriate loss function in a neural network, we must take efficiency into account. For two $p$ dimensional random variables, let the number of samples for the empirical estimate of DC be $n$. Observe that the total cost for computing $\left(a_{k,l}\right)$ is $O\left(n^{2}p\right)$, and the memory to store the intermediate matrices is also $O\left(n^2\right)$. So, we use a stochastic estimate of DC by averaging over minibatches, with each minibatch containing $m$ samples. We describe why this approximation is sensible.

Notation.

We use ${\Theta }_X$, ${\Theta }_Y$ to denote the parameters of the neural networks, and $X$, $Y$ as features extracted by the respective neural networks. Let the minibatch size be $m$, and the dataset $\mathcal{D}=\left({\mathcal{D}}_{\mathcal{X}},{\mathcal{D}}_{\mathcal{Y}}\right)$ be of size $n$. We use ${\left(x_t,y_t\right)}_{t=1}^T$, $x_t\subset {\mathcal{D}}_{\mathcal{X}}$, $y_t\subset {\mathcal{D}}_{\mathcal{Y}}$ to represent the data samples at step $t$, $T$ is the total number of training steps. The distance matrices $A_t$, $B_t$ are computed when given $X_t$, $Y_t$ using (1), which is of dimension $m\hspace{0.17em}\hspace{0.17em}\times \hspace{0.17em}\hspace{0.17em}m$ for each minibatch. Further, we use ${\left(X_t\right)}_k$ to represent the k^th element in $X_t$. And ${\left(A_t\right)}_{k,l}$ is the k^th row and l^th column element in the matrix $A_t$. The inner-product between two matrices $A$, $B$ is defined as $〈A,B〉={\sum }_{i,j}^m{(A)}_{i,j}{(B)}_{i,j}$.

Objective function.

Consider the case where we minimize DC between two networks ${\Theta }_X$, ${\Theta }_{\mathrm{Y.}}$ Since the parameters between ${\Theta }_X$, ${\Theta }_Y$ are separable, we can use the block stochastic gradient iteration in [68] with some simple modifications.

To minimize the distance correlation, we need to solve the following problem

$$\begin{array}{c}\underset{{\Theta }_X,{\Theta }_Y}{\text{min}}\frac{〈A\left({\Theta }_X;x\right),B\left({\Theta }_Y;y\right)〉}{\sqrt{〈A\left({\Theta }_X;x\right),A\left({\Theta }_X;x\right)〉〈B\left({\Theta }_Y;y\right),B\left({\Theta }_Y;y\right)〉}}\\ {(A)}_{k,l}={‖{(X)}_k-{(X)}_l‖}_2,X={\Theta }_X\left(x\right),{(B)}_{k,l}={‖{(Y)}_k-{(Y)}_l‖}_2,Y={\Theta }_Y\left(y\right)\end{array}$$ (5)

We slightly abuse the notation of ${\Theta }_X\left(x\right)$ as applying the network ${\Theta }_X$ onto data $x$, and reuse $A$ to simplify the notation $A\left({\Theta }_X;x\right)$ and the distance matrix. We can rewrite the expression (with $A$, $B$ defined above) using:

$$\underset{{\Theta }_X,{\Theta }_Y}{\text{min}}〈A,B〉\hspace{0.17em}\hspace{0.17em}\text{s}.t. \hspace{0.17em}\hspace{0.17em}\underset{x\subset {\mathcal{D}}_X}{\text{max}}〈A,A〉\le m;\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\underset{y\subset {\mathcal{D}}_{\mathcal{Y}}}{\text{max}}〈B,B〉\le m$$ (6)

where $\left(x,y\right)$ are the minibatch of samples from the data space $\left({\mathcal{D}}_{\mathcal{X}},{\mathcal{D}}_{\mathcal{Y}}\right)$.

We can rewrite the above into the following equation similar to (1) in [68].

$$\underset{{\Theta }_X,{\Theta }_Y}{\text{min}}\Phi \left({\Theta }_X,{\Theta }_Y\right)={\mathbb{E}}_{x,y}f\left({\Theta }_X,{\Theta }_Y;x,y\right)+\gamma \left({\Theta }_X\right)+\gamma \left({\Theta }_Y\right)$$ (7)

where $f\left({\Theta }_X,{\Theta }_Y;x,y\right)$ is $〈A,B〉$ and $\gamma \left({\Theta }_X\right)$ encodes the convex constraint of network ${\Theta }_X\hspace{0.17em}\text{:}\hspace{0.17em}\hspace{0.17em}{\text{max}}_{x\subset {\mathcal{D}}_{\mathcal{X}}}〈A,A〉\le m$. Similarly, $\gamma \left({\Theta }_Y\right)$ encodes ${\text{max}}_{y\subset {\mathcal{D}}_{\mathcal{Y}}}〈B,B〉\le m$. $\Phi \left({\Theta }_X,{\Theta }_Y\right)$ is the constrained objective function to be optimized.

Block stochastic gradient iteration.

We adjust Alg. 1 from [68] to our case in Alg. 1. Since we will need the entire minibatch $\left(x_t,y_t\right)$ to compute the objective function, there will be no mean term when computing the sample gradient ${\tilde{g}}_X^t$. Further, since both blocks $\left({\Theta }_X,{\Theta }_Y\right)$ are constrained, line 3, 5 will use (5) from [68]. The detailed algorithm is presented in Alg. 1.

$$\begin{array}{l}\overline{\underset{¯}{\mathbf{\text{Algorithm}}1\text{ Block Stochastic Gradient for Updating Distance Correlation}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}}}\\ \mathbf{\text{Input}}:\text{ Two neural network with starting point}\hspace{0.17em}{\Theta }_X^1,\hspace{0.17em}{\Theta }_Y^1.\hspace{0.17em}\hspace{0.17em}\text{Training}\hspace{0.17em}\text{data}\hspace{0.17em}{\left\{\left(x_t,y_t\right)\right\}}_{t=1}^T,\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\text{step}\hspace{0.17em}\text{size}\hspace{0.17em}{\eta }_X,{\eta }_Y,\hspace{0.17em}\hspace{0.17em}\text{and}\hspace{0.17em}\text{batch}\hspace{0.17em}\text{size}\hspace{0.17em}m.\\ \mathbf{\text{Output}}:\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{\tilde{\Theta }}_X^T,{\tilde{\Theta }}_Y^T\\ \hspace{0.17em}\hspace{0.17em}1:\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\mathbf{\text{for}}\hspace{0.17em}\hspace{0.17em}t\hspace{0.17em}\text{=}\hspace{0.17em}\text{1,}\dots \text{,}T\hspace{0.17em}\hspace{0.17em}\mathbf{\text{do}}\\ \hspace{0.17em}\hspace{0.17em}\text{2:}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\text{Compute}\hspace{0.17em}\hspace{0.17em}\text{sample}\hspace{0.17em}\hspace{0.17em}\text{gradient}\hspace{0.17em}\hspace{0.17em}\text{for}\hspace{0.17em}\hspace{0.17em}{\Theta }_X\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{\tilde{g}}_X^t={\nabla }_{{\Theta }_X}f\left({\Theta }_X^t,{\Theta }_Y^t;x_t,y_t\right)\\ \hspace{0.17em}\hspace{0.17em}3:\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{\Theta }_X^{t+1}=\arg \hspace{0.17em}{\min }_{{\Theta }_X}〈{\tilde{g}}_X^t+{\tilde{\nabla }}_{\gamma X}\left({\Theta }_X^t\right),{\Theta }_X-{\Theta }_X^t〉+\frac{1}{2_{\eta X}}{‖{\Theta }_X-{\Theta }_X^t‖}^2\\ \hspace{0.17em}\hspace{0.17em}4:\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\text{Compute}\hspace{0.17em}\text{sample gradient}\hspace{0.17em}\text{for}\hspace{0.17em}{\Theta }_Y\\ \hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{\tilde{g}}_Y^t={\nabla }_{{\Theta }_Y}f\left({\Theta }_X^{t+1},{\Theta }_Y^t;x_t,y_t\right)\\ \hspace{0.17em}\hspace{0.17em}5:\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{\Theta }_Y^{t+1}=\arg \hspace{0.17em}{\min }_{{\Theta }_Y}〈{\tilde{g}}_Y^t+{\tilde{\nabla }}_{\gamma Y}\left({\Theta }_Y^t\right),{\Theta }_Y-{\Theta }_Y^t〉+\frac{1}{2_{\eta Y}}{‖{\Theta }_Y-{\Theta }_Y^t‖}^2\\ \hspace{0.17em}\hspace{0.17em}6:\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\mathbf{\text{end}}\hspace{0.17em}\mathbf{\text{for}}\\ \hspace{0.17em}\hspace{0.17em}\text{7:}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{\tilde{\Theta }}_X^T=\frac{1}{T}{\sum }_{t=1}^T{\Theta }_X^t\\ \hspace{0.17em}\hspace{0.17em}\underset{¯}{8:\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}{\tilde{\Theta }}_Y^T=\frac{1}{T}{\sum }_{t=1}^T{\Theta }_Y^t\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}}\end{array}$$

Proposition 1.

After $T$ iterations of Algorithm 1 with step size ${\eta }_X={\eta }_Y=\frac{\eta }{\sqrt{T}}<\frac{1}{L}$, for some positive constant $\eta <\frac{1}{L}$, where $L$ is the Lipschitz constant of the partial gradient of $f$, by Theorem. 6 in [68], we know there exists an index subsequence $\mathcal{T}$ such that:

$$\underset{t\to \infty ,t\in \mathcal{T}}{\text{lim}}\mathbb{E}\left[dist\left(0,\nabla \Phi \left({\Theta }_X^t,{\Theta }_Y^t\right)\right)\right]=0$$ (8)

where dist$\left(\text{y},\mathcal{X}\right)={\text{min}}_{\text{x}\in \mathcal{X}}\parallel \text{x}-\text{y}\parallel$.

But empirically, we find that simply applying Stochastic Gradient Decent (SGD) is sufficient, but this choice is available to the user.

4. Independent Features Help Robustness

Goal.

We show how distance correlation can help us train multiple deep networks that learn mutually independent features, roughly similar to finding diverse $M$-best solutions in structured SVM models [59]. We describe how such an approach can lead to better robustness against adversarial attacks.

Rationale.

Recently, several efforts have explored generating of adversarial examples that can transfer to different networks and how to defend against such attacks [15,62,6]. It is often observed that an adversarial sample for one trained network is relatively easy to transfer to another network with the same architecture [15]. Here, we show that even for as few as two networks (same architecture; trained on the same data), we can, to some extent, prevent adversarial examples from transferring between them by seeking independent features.

Setup.

We formulate the problem considering a classification task as an example. Given two deep neural networks with the same architecture denoted as $f_1\left(\cdot \right)$, $f_2\left(\cdot \right)$, we train them using image-label pairs $\left(x,y\right)$ using the cross-entropy loss ${\text{Loss}}_{\text{CE}}$. If we train $f_1$ and $f_2$ using only the cross-entropy loss, the adversarial examples generated on $f_1$ can relatively easily transfer to $f_2$ (see the performance of “Baseline” in Table 1). To enforce $f_1$ and $f_2$ to learn independent features, let the extracted feature of $x$ in some intermediate layer of $f$ be given as $g\left(x\right)$ (in this section we use the feature before the last fully connected layer as an example). We can still train $f_1$ using ${\text{Loss}}_{\text{CE}}$, and then, we train $f_2$ using,

$${\text{Loss}}_{\text{total }}={\text{Loss}}_{\text{CE}}\left(f_2\left(x\right),y\right)+\alpha \cdot {\text{Loss}}_{\text{DC}}\left(g_1\left(x\right),g_2\left(x\right)\right)$$ (9)

Table 1:

The test accuracy (%) of a model $f_2$ on the adversarial examples generated using $f_1$ with the same architecture. “Baseline”: train without constraint. “Ours”: $f_2$ is independent to $f_1$. “Clean”: test accuracy without adversarial examples.

Dataset	Network	Method	Clean	${\text{FGM}}_{ϵ=0.03}$	${\text{PGD}}_{ϵ=0.03}$	${\text{FGM}}_{ϵ=0.05}$	${\text{PGD}}_{ϵ=0.05}$	${\text{FGM}}_{ϵ=0.1}$		${\text{PGD}}_{ϵ=0.1}$
CIFAR10	Resnet 18	Baseline	89.14	72.10	x66.34	62.00	49.42	48.23	27.41
CIFAR10	Resnet 18	Ours	87.61	74.76	72.85	65.56	59.33	50.24	36.11
ImageNet	Mobilenet-v3-small	Baseline	47.16	29.64	30.00	23.52	24.81	13.90	17.15
ImageNet	Mobilenet-v3-small	Ours	42.34	34.47	36.98	29.53	33.77	19.53	28.04
ImageNet	Efficientnet-B0	Baseline	57.85	26.72	28.22	18.96	19.45	12.04	11.17
ImageNet	Efficientnet-B0	Ours	55.82	30.42	35.99	22.05	27.56	14.16	17.62
ImageNet	Resnet 34	Baseline	64.01	52.62	56.61	45.45	51.11	33.75	41.70
ImageNet	Resnet 34	Ours	63.77	53.19	57.18	46.50	52.28	35.00	43.35
ImageNet	Resnet 152	Baseline	66.88	56.56	59.19	50.61	53.49	40.50	44.49
ImageNet	Resnet 152	Ours	68.04	58.34	61.33	52.59	56.05	42.61	47.17

where $\alpha$ is a constant scalar and ${\text{Loss}}_{\text{DC}}$ is the distance correlation from Def. 1 Note that we do not require $g_1\left(x\right)$ and $g_2\left(x\right)$ to be in the same dimension, so in principle we could easily use features from different layers for these two networks.

Experimental settings.

We first conduct experiments on CIFAR10 [39] using Resnet 18 [28]. We then use four different architectures (mobilenet-v3-small [32] , efficientnet-B0 [66], Resnet 34, and Resnet152) and train them on ImageNet [40]. For each network architecture, we first train two networks using only ${\text{Loss}}_{\text{CE}}$. Next, we train a network using only ${\text{Loss}}_{\text{CE}}$ before training a second network using the loss in (9). On CIFAR10, we utilize the SGD optimizer with momentum 0.9 and train for 200 epochs using an initial learning rate 0.1 with a cosine learning rate scheduler [52]. The mini-batch size is set to 128. On ImageNet [40], we train for 40 epochs using an initial learning rate 0.1, which decays by 0.1 every 10 epochs. The mini-batch size is 512 . Our $\alpha$ in (9) is set to 0.05 for all cases. For each combination of the dataset and the network architecture, we train two networks $f_1$ and $f_2$, after which we generate adversarial examples on $f_1$ and use them to attack $f_2$ and measure its classification accuracy. We construct a baseline by training $f_1$ and $f_{2\text{ Baseline }}$ without constraints. And train $f_{2\text{ Our }}$ using (9) to learn independent features w.r.t. $f_1$. We report performance under two widely used attack methods: fast gradient sign method (FGM) [26] and projected gradient descent method (PGD) [46], where the latter is considered among the strongest attacks. The scale $ϵ$ of the adversarial perturbation is chosen from {0.03, 0.05, 0.1} and the maximum number of iterations of PGD is set to 40.

Results.

The results are shown in Table 1. We see that we get significant improvement in accuracy over the baseline under adversarial attacks, with comparable performance on clean inputs. Notably, our method achieves more than 10% absolute improvement in accuracy under PGD attack on Resnet-18 and Mobilenet-v3-small. This provides evidence supporting the benefits of enforcing the networks to learn independent features using our distance correlation loss.

In Fig. 2, we show correlation results using Picasso [29,7] to lower the dimension of features for each network. The embedding dimension is 2 for visualization. In Fig. 2(a), we show the embedding of different networks. $f_1$ represents the network to generate the adversarial examples. $f_{2\text{ Baseline }}$ denotes the baseline network, trained without distance correlation constraint. Also, $f_{2_{Ours}}$ is the same network trained to be independent to $f_1$. In Fig. 2(b), we visualize the correlation between $f_1$ and $f_{2_{Baseline}}$ for each dimension, and the correlation between $f_1$ and $f_{2_{Ours}}$. If the scatter plot looks circle-like, we can infer that the two models are independent. We see that in different networks, the use of DC shows stronger independence. From Fig. 2/Tab. 1, we also see that the more independent the models are, the better is the gain for transferred attack robustness.

Fig. 2:

Picasso visualization of features space and the correlation between different models. (a) Feature space distribution. (b) Cross-correlation between the feature space of $f_1$ and $f_2$ trained with/without DC. We get better independence. (c) By increasing the balance parameter $\alpha$ of DC loss, Mobilenet is more independent to $f_1$.

5. Informative Comparisons between Networks

Overview.

As discussed in §1, there is much interest in understanding whether two different models learn similar concepts from the data – for example, whether vision Transformers “see” similar to convolutional neural networks [56]. Here, we first follow [56] and discuss similarities between different layers of ViT and ResNets using distance correlation. Next, we investigate that after taking out the influence of Resnets from ViT (or vice versa), what are the residual learned concepts remaining in the network.

5.1. Measure Similarity between Neural Networks

Goal.

We first want to understand whether ViTs represent features across all layers differently from CNNs (such as Resnets). However, analyzing the features in the hidden layers can be challenging, because the features are spread across neurons. Also, different layers have different numbers of neurons. Recently, [56] applied the Centered Kernel Alignment (CKA) for this task. CKA is effective because it involves no constraint on the number of neurons. It is also independent to the orthogonal transformations of representations. Here, we want to demonstrate that distance correlation is a reasonable alternative for CKA in these settings.

Experimental settings.

First, as described in [56], we show that similarity between layers within a single neural network can be assessed using distance correlation (see Fig. 3(a) ). We pick ViT Base with patch 16, and three commonly used Resnets. All networks are pretrained on ImageNet. For ViT, we pick the embedding layer and all the normalization, attention, and fully connected layers within each block. The total number of layers is 63. For Resnets, we use all convolutional layers and the last fully connected layer, which is the same counting method to build Resnet models.

Fig. 3:

(a) Left 4: Similarity between layers within one single model. ViT can be split into small blocks and the similarity from shallow layers to the deeper layers is higher. Most Resnet models show few large blocks in the network, and the last few layers share minimal similarity with the shallow layers. (b) Right 3: Similarity between layers across ViT and Resnets. In the initial 1/6 layers (highlighted in green), the two networks share high similarity. And the last few layers share the least similarity

Results (a).

Our findings add to those from [56]. Using distance correlation, we find that the ViT layers can be split into small blocks and the similarity between different blocks from shallow layers to the deeper layers is higher. For most Resnets, the feature similarity shows that there are a few large blocks in the network, which contains more than 30 layers each, and the last few layers share minimal similarity with the shallow layers.

Results (b).

After within-model distance correlation, we perform across-model distance correlation comparisons between ViT and Resnets, see Fig. 3(b). We notice that in the initial 1/6 layers, the two networks share high similarities. But later, the similarity spreads across all different layers between ViT and Resnets. Notably, the last few layers share the least similarity between two networks.

By using the distance correlation to calculate the heatmap of the similarity matrices, we can qualitatively describe the difference between the patterns of the features in different layers from different networks. What is even more interesting is to quantitatively show the difference, for example, to answer which network contains more information for the ground truth classes. We discuss this next.

5.2. What Remains When “Taking out” $Y$ from $X$

Goal.

Even measuring information contained in one neural network is challenging, and often tackled by measuring the accuracy on the test dataset. But the association between accuracy and the information contained in a network may be weak. Based on existing literature, conditioning one network w.r.t. another remains unresolved. Despite the above challenges, we can indeed measure the similarity between the features of the network $X$ and the ground truth labels. If the similarity is higher, we can say that the feature space of $X$ contains more information regarding the true labels. Distance correlation enables this. Interestingly, partial distance correlation extends this idea to multiple networks allowing us to approach the “conditioning” question posed above.

Rationale/setup.

Here, we choose the last layer before the final fully-connected layer as the feature layer similar to the setup in §4. Our first attempt involved directly applying the distance correlation measurement to feature $X$ and the one-hot ground truth embedding. However, the one-hot embedding for the label contains very little information, e.g., it does not show the difference between “cat” vs. “dog” and “cat” vs. “airplane”. So, we use the pretrained BERT [16] to linguistically embed the class labels into the hidden space. We then measure the distance correlation between the feature space of $X$ and the pretrained hidden space $GT. {\mathcal{R}}^2\left(X,GT\right)=\frac{m}{n}{\sum }_{t=1}^{n/m}dCor\left(x_t,g{t}_t\right)$ where $x_t$ is the feature for one minibatch, and $g{t}_t$ is the BERT embedding vector of the corresponding label. To further extend this metric to measure the “remaining” or residual information, we apply the partial distance correlation calculation by removing $Y$ out of $X$, or say $X$ conditioned on $Y$. Then, we have ${\mathcal{R}}^2\left(\left(X\mid Y\right),GT\right)=\frac{m}{n}{\sum }_{t=1}^{n/m}dCor\left(\left(x_t\mid y_t\right),g{t}_t\right)$ using (4). This capability has not been shown before.

Experimental settings.

In order to measure the information remaining when conditioning network ${\Theta }_Y$ out of ${\Theta }_X$, we first use pretrained networks on ImageNet. We use the validation set of the ImageNet for evaluation. We want to evaluate which network contains the richest information regarding linguistic embedding. Interestingly, we can go beyond such an evaluation, instead, asking the network ${\Theta }_X$ to learn concepts above and beyond what the network ${\Theta }_Y$ has learned. To do so, we include the partial distance correlation into the loss. Unlike the experiment discussed above (minimizing distance correlation), in this setup, we seek to maximize partial distance correlation. The ${\text{Loss}}_{\text{total}}$ is

$${\text{Loss}}_{\text{CE}}\left(f_1\left(x\right),y\right)-\alpha \cdot {\text{Loss}}_{\text{PDC}}\left(\left(g_1\left(x\right)\mid g_2\left(x\right)\right),gt\right)$$ (10)

We take pretrained networks ${\Theta }_X$, ${\Theta }_Y$ and then finetune ${\Theta }_X$ using (10). The learning rate is set to be $1e-5$ and $\alpha$ in the loss term is 1. To check the benefits of partial DC, we use Grad-CAM [60] to highlight the areas that each network is looking at, together with what ${\Theta }_X$ conditioned on ${\Theta }_Y$ sees then.

Results (a).

We first show information comparison between two networks. The details of DC and partial DC are shown in Table. 2. The reader will notice that since ViT achieves the best test accuracy, it also contains the most information. Additionally, although better test accuracy normally coincides with more information, this is not always true. Resnet 50 contains more linguistic information than the much deeper Resnet 152, perhaps a compensation mechanism. For Resnet 152, the network is deep enough to focus on local structures that overwhelm the linguistic information (or this information is unnecessary). This experiment suggests a new strategy to compare two networks beyond test accuracy.

Table 2:

Partial DC between the network ${\Theta }_X$ conditioned on the network ${\Theta }_Y$, and the ImageNet class name embedding. The higher value indicates the more information.

Network ${\Theta }_X$	Network ${\Theta }_Y$	${\mathcal{R}}^2\left(X,GT\right)$	${\mathcal{R}}^2\left(Y,GT\right)$	${\mathcal{R}}^2\left(\left(X\mid Y\right),GT\right)$	${\mathcal{R}}^2\left(\left(Y\mid X\right),GT\right)$
ViT1	Resnet 182	0.042	0.025	0.035	0.007
ViT	Resnet 503	0.043	0.036	0.028	0.017
ViT	Resnet 1524	0.044	0.020	0.040	0.009
ViT	VGG 19 BN5	0.042	0.037	0.026	0.015
ViT	Densenet1216	0.043	0.026	0.035	0.007
ViT large7	Resnet 18	0.046	0.027	0.038	0.007
ViT large	Resnet 50	0.046	0.037	0.031	0.016
ViT large	Resnet 152	0.046	0.021	0.042	0.010
ViT large	ViT	0.045	0.043	0.019	0.013
ViT+Resnet 508	Resnet 18	0.044	0.024	0.037	0.005
Resnet 152	Resnet 18	0.019	0.025	0.013	0.020
Resnet 152	Resnet 50	0.021	0.037	0.003	0.030
Resnet 50	Resnet 18	0.036	0.025	0.027	0.008
Resnet 50	VGG 19 BN	0.036	0.036	0.020	0.019

Accuracy: 1.84.40%; 2. 69.76%; 3. 79.02%; 4. 82.54%; 5. 74.22%; 6. 75.57%; 7. 85.68%; 8. 84.13%

Results (b).

After using a pretrained network, we can also check that by including the partial distance correlation in the loss, which regions does the model pay attention to, using Grad-CAM. We replace the loss term of Grad-CAM with the partial distance correlation. The results are shown in Fig. 4. We see that the pretrained ViT sees across the whole image in different locations, while the Resnet (VGG) tends to focus on only one area of the image. After training, ViT (conditioned on Resnet) pays more attention to the subjects, especially locations outside the Resnet focus. Such experiments help understand how ViT learns beyond Resnets (CNNs).

Fig. 4:

Grad-CAM results on ImageNet using ViT, Resnet18 and VGG16. After using Partial DC to remove the information learned by another network, ViT can focus on detail places and Resnet can only look in major spots. Similar issue happens to VGG.

6. Disentanglement

Overview.

This experiment studies disentanglement [30,36,8,43,21]. It is believed that the image data are generated from low dimensional latent variables – but isolating and disentangling the latent variables is challenging. A key in disentangled latent variable learning is to make the factors in the latent variables independent [2]. Distance correlation fits perfectly and can handle a variety of dimensions for the latent variables. When the distance correlation is 0, we know that the two variables are independent.

Experimental settings.

We follow [21] which focuses on semi-supervised disentanglement to generate high-resolution images. In [21], one divides the latent variables into two categories: (a) attributes of interest – a set of semantic and interpretable attributes, e.g. hair color and age; (b) residual attributes – the remaining information. Formally, $x_i=G\left(f_i^1,\dots ,f_i^k,r_i\right)$, where $G$ is the generator that uses the factors of interest $f_i^l$ and the residual to generate image $x_i$.

In order to enforce the condition that the information regarding the attributes of interest is not leaking into the residual representations, the authors of [21] introduced the loss ${\text{L}}_{\text{res}}={\sum }_{i=1}^n‖r_i{ }^2‖$ to limit the residual information. This is sub-optimal as there can be cases where $r_i$ is not 0 but still independent to the factors of interest ${\left(f_i^l\right)}_{l=1}^k$. Thus, we use distance correlation to replace this loss:

$${\text{L}}_{\text{res}}=dCor\left(\left[f^1;f^2;\dots ;f^k\right],r\right)$$ (11)

We use the same structure proposed in [21], while the generator architecture is adopted from StyleGAN2 [35]. The dataset is the human face dataset FFHQ [34], and the attributes are: age, gender, etc. We use CLIP [54] to partially label the attributes to generate the semi-supervised dataset for training. All losses from [21] are used, except that ${\text{L}}_{\text{res }}$ is replaced by (11).

Results.

(Shown in Fig. 5) Our model shows the ability to change specific attributes without affecting residual features, such as posture (also see supplement).

Fig. 5:

Representative generated images using our training on FFHQ. Note that these results only use semi-supervised dataset by CLIP. Our methods shows the ability to disentangle the attributes of interest and the remaining information.

7. Conclusions

In this paper, we studied how distance correlation (and partial distance correlation) has a wide variety of uses in deep learning tasks in vision. The measure offers various properties that are often enforced using alternative means, that are often far more involved. Further, it is extremely simple to incorporate in contrast to various divergence-based measures often used in invariant representation learning. Notably, the use of partial distance correlation offers the ability of conditioning, which is underexplored in the community. We showcase three very different settings, ranging from network comparison to training distinct/different models to disentanglement where the idea is immediately beneficial, and expect that numerous other applications will emerge in short order.

1.

Download video file ^{(56.1MB, mp4)}