The Convex Information Bottleneck Lagrangian

Abstract

The information bottleneck (IB) problem tackles the issue of obtaining relevant compressed representations T of some random variable X for the task of predicting Y. It is defined as a constrained optimization problem that maximizes the information the representation has about the task, $I(T;Y)$, while ensuring that a certain level of compression r is achieved (i.e., $I(X;T)\le r$). For practical reasons, the problem is usually solved by maximizing the IB Lagrangian (i.e., ${\mathcal{L}}_{\mathrm{IB}}(T;\beta )=I(T;Y)-\beta I(X;T)$) for many values of $\beta \in [0,1]$. Then, the curve of maximal $I(T;Y)$ for a given $I(X;T)$ is drawn and a representation with the desired predictability and compression is selected. It is known when Y is a deterministic function of X, the IB curve cannot be explored and another Lagrangian has been proposed to tackle this problem: the squared IB Lagrangian: ${\mathcal{L}}_{\mathrm{sq}-\mathrm{IB}}(T;{\beta }_{\mathrm{sq}})=I(T;Y)-{\beta }_{\mathrm{sq}}I{(X;T)}^2$. In this paper, we (i) present a general family of Lagrangians which allow for the exploration of the IB curve in all scenarios; (ii) provide the exact one-to-one mapping between the Lagrange multiplier and the desired compression rate r for known IB curve shapes; and (iii) show we can approximately obtain a specific compression level with the convex IB Lagrangian for both known and unknown IB curve shapes. This eliminates the burden of solving the optimization problem for many values of the Lagrange multiplier. That is, we prove that we can solve the original constrained problem with a single optimization.

1. Introduction

Let $X\in \mathcal{X}$ and $Y\in \mathcal{Y}$ be two statistically dependent random variables with joint distribution $p_{(X,Y)}$. The information bottleneck (IB) [1] investigates the problem of extracting the relevant information from X for the task of predicting Y.

For this purpose, the IB defines a bottleneck variable $T\in \mathcal{T}$ obeying the Markov chain $Y\leftrightarrow X\leftrightarrow T$ so that T acts as a representation of X. Tishby et al. [1] define the relevant information as the information the representation keeps from Y after the compression of X (i.e., $I(T;Y)$), provided a certain level of compression (i.e., $I(X;T)\le r$). Therefore, we select the representation which yields the value of the IB curve that best fits our requirements.

Definition 1 (IB Functional).

Let X and Y be statistically dependent variables. Let Δ be the set of random variables T obeying the Markov condition $Y\leftrightarrow X\leftrightarrow T$. Then the IB functional is

$$F_{\mathrm{IB},\max }\left(r\right)=\underset{T\in \Delta }{max}\left\{I(T;Y)\right\}s.t.I(X;T)\le r,\forall r\in [0,\infty ).$$ (1)

Definition 2 (IB Curve).

The IB curve is the set of points defined by the solutions of$F_{\mathrm{IB},\max }\left(r\right)$for varying values of$r\in [0,\infty )$.

Definition 3 (Information Plane).

The plane is defined by the axes $I(T;Y)$ and $I(X;T)$.

This method has been successfully applied to solve different problems from a variety of domains. For example:

• Supervised learning. In supervised learning, we are presented with a set of n pairs of input features and task outputs instances. We seek an approximation of the conditional probability distribution between the task outputs Y and the input features X. In classification tasks (i.e., when Y is a discrete random variable), the introduction of the variable T learned through the information bottleneck principle maintained the performance of standard algorithms based on the cross-entropy loss while providing with more adversarial attacks robustness and invariance to nuisances [2,3,4]. Moreover, by the nature of its definition the information bottleneck appears to be closely related with a trade-off between accuracy on the observable set and generalization to new, unseen instances (see Section 2).

• Clustering. In clustering, we are presented with a set of n pairs of instances of a random variable X and their attributes of interest Y. We seek groups of instances (or clusters T) such that the attributes of interest within the instances of each cluster are similar and the attributes of interest of the instances of different clusters are dissimilar. Therefore, the information bottleneck can be employed since it allows us to aim for attribute representative clusters (maximizing the similarity between instances within the clusters) and enforce a certain compression of the random variable X (ensuring a certain difference between instances of the different clusters). This has been successfully implemented, for instance, for gene expression analysis and word, document, stock pricing, or movie rating clustering [5,6,7].

• Image segmentation. In image segmentation, we want to partition an image into segments such that each pixel in a region shares some attributes. If we divide the image into very small regions X (e.g., each region is a pixel or a set of pixels defined by a grid), we can consider the problem of segmentation as that of clustering the regions X based on the region attributes Y. Hence, we can use the information bottleneck so that we seek region clusters T that are maximally informative about the attributes Y (e.g., the intensity histogram bins) and maintain a level of compression of the original regions X [8].

• Quantization. In quantization, we consider a random variable $X\in \mathcal{X}$ such that $\mathcal{X}$ is a large or continuous set. Our objective is to map X into a variable $T\in \mathcal{T}$ such that $\mathcal{T}$ is a smaller, countable set. If we fix the quantization set size to $|\mathcal{T}|=\lfloor r\rfloor$ and aim at maximizing the information of the quantized variable with another random variable Y and restrict the mapping to be deterministic, then the problem is equivalent to the information bottleneck [9,10].

• Source coding. In source coding, we consider a data source $\mathcal{S}$ which generates a signal $Y\in \mathcal{Y}$, which is later perturbed by a channel $\mathcal{C}:\mathcal{Y}\to \mathcal{X}$ that outputs X. We seek a coding scheme that generates a code $T\in \mathcal{T}$ from the output of the channel X which is as informative as possible about the original source signal Y and can be transmitted at a small rate $I(X;T)\le r$. Therefore, this problem is equivalent to the the formulation of the information bottleneck [11].

Furthermore, it has been employed as a tool for development or explanation in other disciplines like reinforcement learning [12,13,14], attribution methods [15], natural language processing [16], linguistics [17] or neuroscience [18]. Moreover, it has connections with other problems such as source coding with side information (or the Wyner-Ahlswede-Körner (WAK) problem), the rate-distortion problem or the cost-capacity problem (see Sections 3, 6 and 7 from [19]).

In practice, solving a constrained optimization problem such as the IB functional is challenging. Thus, in order to avoid the non-linear constraints from the IB functional, the IB Lagrangian is defined.

Definition 4 (IB Lagrangian).

Let X and Y be statistically dependent variables. Let Δ be the set of random variables T obeying the Markov condition $Y\leftrightarrow X\leftrightarrow T$. Then we define the IB Lagrangian as

$${\mathcal{L}}_{\mathrm{IB}}(T;\beta )=I(T;Y)-\beta I(X;T).$$ (2)

Here $\beta \in [0,1]$ is the Lagrange multiplier which controls the trade-off between the information of Y retained and the compression of X. Note we consider $\beta \in [0,1]$ because (i) for $\beta \le 0$ many uncompressed solutions such as $T=X$ maximize ${\mathcal{L}}_{\mathrm{IB}}(T;\beta )$, and (ii) for $\beta \ge 1$ the IB Lagrangian is non-positive due to the data processing inequality (DPI) (Theorem 2.8.1 from Cover and Thomas [20]) and trivial solutions like $T=\mathrm{const}$ are maximizers with ${\mathcal{L}}_{\mathrm{IB}}(T;\beta )=0$ [21].

We know the solutions of the IB Lagrangian optimization (if existent) are solutions of the IB functional by the Lagrange’s sufficiency theorem (Theorem 5 in Appendix A of Courcoubetis [22]). Moreover, since the IB functional is concave (Lemma 5 of Gilad-Bachrach et al. [19]) we know they exist (Theorem 6 in Appendix A of Courcoubetis [22]).

Therefore, the problem is usually solved by maximizing the IB Lagrangian with adaptations of the Blahut-Arimoto algorithm [1], deterministic annealing approaches [23] or a bottom-up greedy agglomerative clustering [6] or its improved sequential counterpart [24]. However, when provided with high-dimensional random variables X such as images, these algorithms do not scale well and deep learning-based techniques, where the IB Lagrangian is used as the objective function, prevailed [2,25,26].

Note the IB Lagrangian optimization yields a representation T with a given performance ($I(X;T),I(T;Y)$) for a given $\beta$. However, there is no one-to-one mapping between $\beta$ and $I(X;T)$. Hence, we cannot directly optimize for the desired compression level r but we need to perform several optimizations for different values of $\beta$ and select the representation with the desired performance; e.g., [2]. The Lagrange multiplier selection is important since (i) sometimes even choices of $\beta <1$ lead to trivial representations such that $p_{T|X}=p_T$, and (ii) there exist some discontinuities on the performance level w.r.t. the values of $\beta$ [27].

Moreover, recently Kolchinsky et al. [21] showed how in deterministic scenarios (such as many classification problems where an input $x_i$ belongs to a single particular class $y_i$) the IB Lagrangian could not explore the IB curve. Particularly, they showed that multiple $\beta$ yielded the same performance level and that a single value of $\beta$ could result in different performance levels. To solve this issue, they introduced the squared IB Lagrangian, ${\mathcal{L}}_{\mathrm{sq}-\mathrm{IB}}(T;{\beta }_{\mathrm{sq}})=I(T;Y)-{\beta }_{sq}I{(X;T)}^2$, which is able to explore the IB curve in any scenario by optimizing for different values of ${\beta }_{\mathrm{sq}}$. However, even though they realized a one-to-one mapping between ${\beta }_{\mathrm{sq}}$ and the compression level existed, they did not find such mapping. Hence, multiple optimizations of the Lagrangian were still required to find the best trade-off solution.

The main contributions of this article are:

1. We introduce a general family of Lagrangians (the convex IB Lagrangians) which are able to explore the IB curve in any scenario for which the squared IB Lagrangian [21] is a particular case of. More importantly, the analysis made for deriving this family of Lagrangians can serve as inspiration for obtaining new Lagrangian families that solve other objective functions with intrinsic trade-offs such as the IB Lagrangian.

2. We show that in deterministic scenarios (and other scenarios where the IB curve shape is known) one can use the convex IB Lagrangian to obtain a desired level of performance with a single optimization. That is, there is a one-to-one mapping between the Lagrange multiplier used for the optimization and the level of compression and informativeness obtained, and we provide the exact mapping. This eliminates the need for multiple optimizations to select a suitable representation.

3. We introduce a particular case of the convex IB Lagrangians: the shifted exponential IB Lagrangian, which allows us to approximately obtain a specific compression level in any scenario. This way, we can approximately solve the initial constrained optimization problem from Equation (1) with a single optimization.

Furthermore, we provide some insight for explaining why there are discontinuities in the performance levels w.r.t. the values of the Lagrange multipliers. In a classification setting, we connect those discontinuities with the intrinsic clusterization of the representations when optimizing the IB bottleneck objective.

The structure of the article is the following: In Section 2 we motivate the usage of the IB in supervised learning settings. Then, in Section 3 we outline the important results used about the IB curve in deterministic scenarios. Later, in Section 4 we introduce the convex IB Lagrangian and explain some of its properties like the bijective mapping between Lagrange multipliers and the compression level and the range of such multipliers. After that, we support our (proved) claims with some empirical evidence on the MNIST [28] and TREC-6 [29] datasets in Section 5. Finally, in Section 6 we discuss our claims and empirical results. A PyTorch [30] implementation of the article can be found at https://github.com/burklight/convex-IB-Lagrangian-PyTorch.

In the Appendix A, Appendix B, Appendix C, Appendix D, Appendix E and Appendix F we provide with the proofs of the theoretical results. Then, in Appendix G we show some alternative families of Lagrangians with similar properties. Later, in Appendix H we provide with the precise experimental setup details to reproduce the results from the paper, and further experimentation with different datasets and neural network architectures. To conclude, in Appendix I we show some guidelines on how to set the convex information bottleneck Lagrangians for practical problems.

2. The IB in Supervised Learning

In this section, we will first give an overview of supervised learning in order to later motivate the usage of the information bottleneck in this setting.

2.1. Supervised Learning Overview

In supervised learning we are given a dataset ${\mathcal{D}}_n={\left\{(x_i,y_i)\right\}}_{i=1}^n$ of n pairs of input features and task outputs. In this case, X and Y are the random variables of the input features and the task outputs. We assume $x_i$ and $y_i$ are sampled i.i.d. from the true distribution $p_{(X,Y)}=p_{Y|X}{p}_X$. The usual aim of supervised learning is to use the dataset ${\mathcal{D}}_n$ to learn a particular conditional distribution $q_{\widehat{Y}|X}$ of the task outputs given the input features, parametrized by $\theta$, which is a good approximation of $p_{Y|X}$. We use $\widehat{Y}$ and $\widehat{y}$ to indicate the predicted task output random variable and its outcome. We call a supervised learning task regression when Y is continuous-valued and classification when it is discrete.

Usually, supervised learning methods employ intermediate representations of the inputs before making predictions about the outputs; e.g., hidden layers in neural networks (Chapter 5 from Bishop [31]) or transformations in a feature space through the kernel trick in kernel machines like SVMs or RVMs (Sections 7.1 and 7.2 from Bishop [31]). Let T be a possibly stochastic function of the input features X with a parametrized conditional distribution $q_{T|X}$, then, T obeys the Markov condition $Y\leftrightarrow X\leftrightarrow T$. The mapping from the representation to the predicted task outputs is defined by the parametrized conditional distribution $q_{\widehat{Y}|T}$. Therefore, in representation-based machine learning methods, the full Markov Chain is $Y\leftrightarrow X\leftrightarrow T\leftrightarrow \widehat{Y}$. Hence, the overall estimation of the conditional probability $p_{Y|X}$ is given by the marginalization of the representations; i.e., $q_{\widehat{Y}|X}={\mathbb{E}}_{t\sim q_{T|X}}\left[q_{\widehat{Y}|T=t}\right]$ (The notation $q_{\widehat{Y}|T=t}$ represents the probability distribution $q_{\widehat{Y}|T}(·|t;\theta )$. For the rest of the text, we will use the same notation to represent conditional probability distributions where the conditioning argument is given).

In order to achieve the goal of having a good estimation of the conditional probability distribution $p_{Y|X}$, we usually define an instantaneous cost function $𝒿:\mathcal{X}\times \mathcal{Y}\to \mathbb{R}$. The value of this function $𝒿(x,y;\theta )$ serves as a heuristic to measure the loss of our algorithm, parametrized by $\theta$, obtains when trying to predict the realization of the task output y with the input realization x.

Clearly, we can be interested in minimizing the expectation of the instantaneous cost function over all the possible input features and task outputs, which we call the cost function. However, since we only have a finite dataset ${\mathcal{D}}_n$ we have instead to minimize the empirical cost function.

Definition 5 (Cost Function and Empirical Cost Function).

Let X and Y be the input features and task output random variables and $x\in \mathcal{X}$ and $y\in \mathcal{Y}$ their realizations. Let also $𝒿$ be the instantaneous cost function, θ the parametrization of our learning algorithm, and ${\mathcal{D}}_n={\left\{(x_i,y_i)\right\}}_{i=1}^n$ the given dataset. Then, we define:

$$\begin{array}{cc}1.Thecostfunction:\hfill & J(p_{(X,Y)};\theta )={\mathbb{E}}_{(x,y)\sim p_{(X,Y)}}\left[𝒿(x,y;\theta )\right]\hfill \end{array}$$ (3)

$$\begin{array}{cc}2.Theempricalcostfunction:\hfill & \widehat{J}({\mathcal{D}}_n;\theta )=\frac{1}{n}\sum _{i=1}^n𝒿(x_i,y_i;\theta )\hfill \end{array}$$ (4)

The discrepancy between the normal and empirical cost functions is called the generalization gap or generalization error (see Section 1 of Xu and Raginsky [32], for instance) and intuitively, the smaller this gap is, the better our model generalizes; i.e., the better it will perform to new, unseen samples in terms of our cost function.

Definition 6 (Generalization Gap).

Let $J(p_{(X,Y)};\theta )$ and $\widehat{J}({\mathcal{D}}_n;\theta )$ be the cost and the empirical cost functions as defined in Definition 5. Then, the generalization gap is defined as

$$\mathrm{gen}({\mathcal{D}}_n;\theta )=J(p_{(X,Y)};\theta )-\widehat{J}({\mathcal{D}}_n;\theta ),$$ (5)

and it represents the error incurred when the selected distribution is the one parametrized by θ when the rule $\widehat{J}({\mathcal{D}}_n;\theta )$ is used instead of $J(p_{(X,Y)};\theta )$ as the function to minimize.

Ideally, we would want to minimize the cost function. Hence, we usually try to minimize the empirical cost function and the generalization gap simultaneously. The modifications to our learning algorithm which intend to reduce the generalization gap but not hurt the performance on the empirical cost function are known as regularization.

2.2. Why Do We Use the IB?

Definition 7 (Representation cross-entropy cost function).

Let X and Y be two statistically dependent variables with joint distribution $p_{(X,Y)}=p_{Y|X}{p}_X$. Let also T be a random variable obeying the Markov condition $Y\leftrightarrow X\leftrightarrow T$ and $q_{T|X}$ and $q_{\widehat{Y}|T}$ be the encoding and decoding distributions of our model, parametrized by θ. Finally, let $\mathbb{C}(p_Z||q_Z)=-{\mathbb{E}}_{z\sim p_Z}[log\left(q_Z\left(z\right)\right)]$ be the cross entropy between two probability distributions $p_Z$ and $q_Z$. Then, the cross-entropy cost function is

$$\begin{array}{c}\hfill J_{\mathrm{CE}}(p_{(X,Y)};\theta )={\mathbb{E}}_{(x,t)\sim q_{T|X}{p}_X}\left[\mathbb{C}(q_{Y|T=t}||q_{\widehat{Y}|T=t})\right]={\mathbb{E}}_{(x,y)\sim p_{(X,Y)}}\left[{𝒿}_{\mathrm{CE}}(x,y;\theta )\right],\end{array}$$ (6)

where ${𝒿}_{\mathrm{CE}}(x,y;\theta )=-{\mathbb{E}}_{t\sim q_{T|X=x}}[q_{\widehat{Y}|T=t}(y|t;\theta )]$ is the instantaneous representation cross-entropy cost function and $q_{Y|T}={\mathbb{E}}_{x\sim p_X}[p_{Y|X=x}{q}_{T|X=x}/q_T]$ and $q_T={\mathbb{E}}_{x\sim p_X}\left[q_{T|X=x}\right]$.

The cross-entropy is a widely used cost function in classification tasks (e.g., Teahan [8], Krizhevsky et al. [33], Shore and Gray [34]) which has many interesting properties [35]. Moreover, it is known that minimizing the $J_{\mathrm{CE}}(p_{(X,Y)};\theta )$ maximizes the mutual information $I(T;Y)$. That is:

Proposition 1 (Minimizing the Cross Entropy Maximizes the Mutual Information).

Let $J_{\mathrm{CE}}(p_{(X,Y)};\theta )$ be the representation cross-entropy cost function as defined in Definition 7. Let also $I(T;Y)$ be the mutual information between random variables T and Y in the setting from Definition 7. Then, minimizing $J_{\mathrm{CE}}(p_{(X,Y)};\theta )$ implies maximizing $I(T;Y)$.

The proof of this proposition can be found in Appendix A.

Definition 8 (Nuisance).

A nuisance is any random variable that affects the observed data X but is not informative to the task we are trying to solve. That is, Ξ is a nuisance for Y if $Y\perp \Xi$ or $I(\Xi ,Y)=0$.

Similarly, we know that minimizing $I(X;T)$ minimizes the generalization gap for restricted classes when using the cross-entropy cost function (Theorem 1 of Vera et al. [36]), and when using $I(T;Y)$ directly as an objective to maximize (Theorem 4 of Shamir et al. [37]). Furthermore, Achille and Soatto [38] in Proposition 3.1 upper bound the information of the input representations, T, with nuisances that affect the observed data, $\Xi$, with $I(X;T)$. Therefore, minimizing $I(X;T)$ helps generalization by not keeping useless information of $\Xi$ in our representations.

Thus, jointly maximizing $I(T;Y)$ and minimizing $I(X;T)$ is a good choice both in terms of performance in the available dataset and in new, unseen data, which motivates studies on the IB.

3. The Information Bottleneck in Deterministic Scenarios

Kolchinsky et al. [21] showed that when Y is a deterministic function of X (i.e., $Y=f\left(X\right)$), the IB curve is piecewise linear. More precisely, it is shaped as stated in Proposition 2.

Proposition 2 (The IB Curve is Piecewise Linear in Deterministic Scenarios).

Let X be a random variable and $Y=f\left(X\right)$ be a deterministic function of X. Let also T be the bottleneck variable that solves the IB functional. Then the IB curve in the information plane is defined by the following equation:

$$\left\{\begin{array}{ccc}I(T;Y)=I(X;T)\hfill & \mathrm{if}\hfill & I(X;T)\in [0,I(X;Y\left)\right)\hfill \\ I(T;Y)=I(X;Y)\hfill & \mathrm{if}\hfill & I(X;T)\ge I(X;Y)\hfill \end{array}\right.$$ (7)

Furthermore, they showed that the IB curve could not be explored by optimizing the IB Lagrangian for multiple $\beta$ because the curve was not strictly concave. That is, there was not a one-to-one relationship between $\beta$ and the performance level.

Theorem 1 (In Deterministic Scenarios, the IB Curve cannot be Explored Using the IB Lagrangian).

Let X be a random variable and $Y=f\left(X\right)$ be a deterministic function of X. Let also Δ be the set of random variables T obeying the Markov condition $Y\leftrightarrow X\leftrightarrow T$. Then:

• 1.  Any solution $T\in \Delta$ such that $I(X;T)\in [0,I(X;Y\left)\right)$ and $I(T;Y)=I(X;T)$ solves ${arg\; max}_{T\in \Delta }\left\{{\mathcal{L}}_{\mathrm{IB}}(T;\beta )\right\}$ for $\beta =1$. That is, many different compression and performance levels can be achieved for $\beta =1$.
• 2.  Any solution $T\in \Delta$ such that $I(X;T)>I(X;Y)$ and $I(T;Y)=I(X;Y)$ solves ${arg\; sup}_{T\in \Delta }\left\{{\mathcal{L}}_{\mathrm{IB}}(T;\beta )\right\}$ for $\beta =0$. That is, many compression levels can be achieved with the same performance for $\beta =0$.

  Note we use the supremum in this case since for $\beta =0$ we have that $I(X;T)$ could be infinite and then the search set from Equation (1); i.e., $\{T:Y\leftrightarrow X\leftrightarrow T\}\cap \{T:I(X;T)<\infty \}$ is not compact anymore.

• 3. Any solution$T\in \Delta$such that$I(X;T)=I(T;Y)=I(X;Y)$solves${arg\; max}_{T\in \Delta }\left\{{\mathcal{L}}_{\mathrm{IB}}(T;\beta )\right\}$for all$\beta \in (0,1)$. That is, many different β achieve the same compression and performance level.

An alternative proof for this theorem can be found in Appendix B.

4. The Convex IB Lagrangian

4.1. Exploring the IB Curve

Clearly, a situation like the one depicted in Theorem 1 is not desirable, since we cannot aim for different levels of compression or performance. For this reason, we generalize the effort from Kolchinsky et al. [21] and look for families of Lagrangians which are able to explore the IB curve. Inspired by the squared IB Lagrangian, ${\mathcal{L}}_{\mathrm{sq}-\mathrm{IB}}(T;{\beta }_{\mathrm{sq}})=I(T;Y)-{\beta }_{\mathrm{sq}}I{(X;T)}^2$, we look at the conditions a function of $I(X;T)$ requires in order to be able to explore the IB curve. In this way, we realize that any monotonically increasing and strictly convex function will be able to do so, and we call the family of Lagrangians with these characteristics the convex IB Lagrangians, due to the nature of the introduced function.

Theorem 2 (Convex IB Lagrangians).

Let Δ be the set of r.v. T obeying the Markov condition $Y\leftrightarrow X\leftrightarrow T$. Then, if u is a monotonically increasing and strictly convex function, the IB curve can always be recovered by the solutions of ${arg\; max}_{T\in \Delta }\left\{{\mathcal{L}}_{\mathrm{IB},u}(T;{\beta }_u)\right\}$, with

$${\mathcal{L}}_{\mathrm{IB},u}(T;{\beta }_u)=I(T;Y)-{\beta }_{u}u\left(I(X;T)\right).$$ (8)

That is, for each point $\left(I\right(X;T),I(T;Y\left)\right)$ s.t. $dI(T;Y)/dI(X;T)>0$ there is a unique ${\beta }_u$ for which maximizing ${\mathcal{L}}_{\mathrm{IB},u}(T;{\beta }_u)$ achieves this solution. Furthermore, ${\beta }_u$ is strictly decreasing w.r.t. $I(X;T)$. We call ${\mathcal{L}}_{\mathrm{IB},u}(T;{\beta }_u)$ the convex IB Lagrangian.

The proof of this theorem can be found in Appendix C. Furthermore, by exploiting the IB curve duality (Lemma 10 of Gilad-Bachrach et al. [19]) we were able to derive other families of Lagrangians which allow for the exploration of the IB curve (Appendix G).

Remark 1.

Clearly, we can see how if u is the identity function (i.e., $u\left(I\right(X;T\left)\right)=I(X;T)$) then we end up with the normal IB Lagrangian. However, since the identity function is not strictly convex, it cannot ensure the exploration of the IB curve.

During the proof of this theorem we observed a relationship between the Lagrange multipliers and the solutions obtained of the normal IB Lagrangian ${\mathcal{L}}_{\mathrm{IB}}(T;\beta )$ and the convex IB Lagrangian ${\mathcal{L}}_{\mathrm{IB},u}(T;{\beta }_u)$. This relationship is formalized in the following corollary.

Corollary 1 (IB Lagrangian and IB convex Lagrangian connection).

Let ${\mathcal{L}}_{\mathrm{IB}}(T;\beta )$ be the IB Lagrangian and ${\mathcal{L}}_{\mathrm{IB},u}(T;{\beta }_u)$ the convex IB Lagrangian. Then, maximizing ${\mathcal{L}}_{\mathrm{IB}}(T;\beta )$ and ${\mathcal{L}}_{\mathrm{IB},u}(T;{\beta }_u)$ can obtain the same point in the IB curve if ${\beta }_u=\beta /u^{\prime }\left(I(X;T)\right)$, where $u^{\prime }$ is the derivative of u.

This corollary allows us to better understand why the addition of u allows for the exploration of the IB curve in deterministic scenarios. If we note that for $\beta =1$ we can obtain any point in the increasing region of the curve, then we clearly see how evaluating $u^{\prime }$ for different values of $I(X;T)$ define different values of ${\beta }_u$ that obtain such points. Moreover, it lets us see how if for $\beta =0$ maximizing the IB Lagrangian could obtain any point $\left(I\right(X;Y);I(X;T\left)\right)$ with $I(X;T)>I(X;Y)$, then the same happens for the IB convex Lagrangian.

4.2. Aiming for a Specific Compression Level

Let $B_u$ denote the domain of Lagrange multipliers ${\beta }_u$ for which we can find solutions in the IB curve with the convex IB Lagrangian. Then, the convex IB Lagrangians do not only allow us to explore the IB curve with different ${\beta }_u$. They also allow us to identify the specific ${\beta }_u$ that obtains a given point $\left(I\right(X;T),I(T;Y\left)\right)$, provided we know the IB curve in the information plane. Conversely, the convex IB Lagrangian allows finding the specific point $\left(I\right(X;T),I(T;Y\left)\right)$ that is obtained by a given ${\beta }_u$.

Proposition 3 (Bijective Mapping between IB Curve Point and Convex IB Lagrange multiplier).

Let the IB curve in the information plane be known; i.e., $I(T;Y)=f_{\mathrm{IB}}\left(I(X;T)\right)$ is known. Then there is a bijective mapping from Lagrange multipliers ${\beta }_u\in B_u\backslash \left\{0\right\}$ from the convex IB Lagrangian to points in the IB curve $(I(X;T),f_{\mathrm{IB}}\left(I(X;T)\right)$. Furthermore, these mappings are:

$${\beta }_u=\frac{d{f}_{\mathrm{IB}}\left(I(X;T)\right)}{dI(X;T)}\frac{1}{u^{\prime }\left(I(X;T)\right)}andI(X;T)={\left(u^{\prime }\right)}^{-1}\left(\frac{d{f}_{\mathrm{IB}}\left(I(X;T)\right)}{dI(X;T)}\frac{1}{{\beta }_u}\right),$$ (9)

where $u^{\prime }$ is the derivative of u and ${\left(u^{\prime }\right)}^{-1}$ is the inverse of $u^{\prime }$.

This is especially interesting since in deterministic scenarios we know the shape of the IB curve (Theorem 2) and since the convex IB Lagrangians allow for the exploration of the IB curve (Theorem 2). A proof for Proposition 3 can be found in Appendix D.

Remark 2.

Note that the definition from Tishby et al. [1] $\beta =d{f}_{\mathrm{IB}}\left(I(X;T)\right)/dI(X;T)$ only allows for a bijection between β and $I(X;T)$ if $f_{\mathrm{IB}}$ is a strictly convex, and known function, and we have seen this is not the case in deterministic scenarios (Theorem 1).

A direct result derived from this proposition is that we know the domain of Lagrange multipliers, $B_u$, which allows for the exploration of the IB curve if the shape of the IB curve is known. Furthermore, if the shape is not known we can at least bound that range.

Corollary 2 (Domain of Convex IB Lagrange Multiplier with Known IB Curve Shape).

Let the IB curve in the information plane be $I(T;Y)=f_{\mathrm{IB}}\left(I(X;T)\right)$ and let $I_{\max }=I(X;Y)$. Let also $I(X;T)=r_{\max }$ be the minimum mutual information s.t. $f_{\mathrm{IB}}\left(r_{\max }\right)=I_{\max }$; i.e., $r_{\max }={arg\; inf}_r\left\{f_{\mathrm{IB}}\left(r\right)\right\}\mathrm{s}.\mathrm{t}.f_{\mathrm{IB}}\left(r\right)=I_{\max }$. Then, the range of Lagrange multipliers that allow the exploration of the IB curve with the convex IB Lagrangian is $B_u=[{\beta }_{u,\min },{\beta }_{u,\max }]$, with

$${\beta }_{u,\min }=\underset{r\to r_{\max }^{-}}{lim}\left\{\frac{f_{\mathrm{IB}}^{\prime }\left(r\right)}{u^{\prime }\left(r\right)}\right\}and{\beta }_{u,\max }=\underset{r\to 0^{+}}{lim}\left\{\frac{f_{\mathrm{IB}}^{\prime }\left(r\right)}{u^{\prime }\left(r\right)}\right\},$$ (10)

where $f_{\mathrm{IB}}^{\prime }\left(r\right)$ and $u^{\prime }\left(r\right)$ are the derivatives of $f_{\mathrm{IB}}\left(I(X;T)\right)$ and $u\left(I\right(X;T\left)\right)$ w.r.t. $I(X;T)$ evaluated at r respectively. Also, note that there are some scenarios where $r_{\max }\to \infty$ (see, e.g., [39]), in these scenarios ${\beta }_{u,\min }={lim}_{r\to \infty }\left\{f_{\mathrm{IB}}^{\prime }\left(r\right)/u^{\prime }\left(r\right)\right\}\ge 0$.

Corollary 3 (Domain of Convex IB Lagrange Multiplier Bound).

The range of the Lagrange multipliers that allow the exploration of the IB curve is contained by $[0,{\beta }_{\mathrm{u},\mathrm{top}}]$ which is also contained by $[0,{\beta }_{u,\mathrm{top}}^{+}]$, where

$${\beta }_{u,\mathrm{top}}=\frac{{\left({inf}_{{\Omega }_x\subset \mathcal{X}}\left\{{\beta }_0\left({\Omega }_x\right)\right\}\right)}^{-1}}{{lim}_{r\to 0^{+}}\left\{u^{\prime }\left(r\right)\right\}},and{\beta }_{u,\mathrm{top}}^{+}=\frac{1}{{lim}_{r\to 0^{+}}\left\{u^{\prime }\left(r\right)\right\}},$$ (11)

where $u^{\prime }\left(r\right)$ is the derivative of $u\left(I\right(X;T\left)\right)$ w.r.t. $I(X;T)$ evaluated at r, $\mathcal{X}$ is the set of possible realizations of X and ${\beta }_0$ and ${\Omega }_x$ are defined as in [27] (Note in [27] they consider the dual problem (see Appendix G), so when they refer to ${\beta }^{-1}$ it translates to β in this article). That is, $B_u\subseteq [0,{\beta }_{u,\mathrm{top}}]\subseteq [0,{\beta }_{u,\mathrm{top}}^{+}]$.

Corollaries 2 and 3 allow us to reduce the range search for $\beta$ when we want to explore the IB curve. Practically, ${inf}_{{\Omega }_x\subset \mathcal{X}}\left\{{\beta }_0\left({\Omega }_x\right)\right\}$ might be difficult to calculate so Wu et al. [27] derived an algorithm to approximate it. However, we still recommend setting the numerator to 1 for simplicity. The proofs for both corollaries are found in Appendix E and Appendix F.

5. Experimental Support

In order to showcase our claims we use the MNIST [28] and the TREC-6 [29] datasets. We modify the nonlinear-IB method [26], which is a neural network that minimizes the cross-entropy while also minimizing a differentiable kernel-based estimate of $I(X;T)$ [40]. Then, we used this technique to maximize a lower bound on the convex IB Lagrangians by applying the functions u to the $I(X;T)$ estimate.

The network structure is the following: first, a stochastic encoder $T=f_{\mathrm{enc}}(X;\theta )+W$ with $p_W=\mathcal{N}(0,I_d)$ such that $T\in {\mathbb{R}}^d$, where d is the dimension of the bottleneck variable (Note that the encoder needs to be stochastic to (i) ensure a finite and well-defined mutual information [21,41] and (ii) make gradient-based optimization methods over the IB Lagrangian useful [41]). Second, a deterministic decoder $q_{\widehat{Y}|T}=f_{\mathrm{dec}}(T;\theta )$. For the MNIST dataset both the encoder and the decoder are fully-connected networks, for a fair comparison with [26]. For the TREC-6 dataset, the encoder is a set of convolutions of word embeddings followed by a fully-connected network and the decoder is also a fully-connected network. For further details about the experiment setup, additional results for different values of $\alpha$ and $\eta$ and supplementary experimental results for different datasets and network architectures, please refer to Appendix H.

In Figure 1 we show our results for two particularizations of the convex IB Lagrangians:

1. the power IB Lagrangians: ${\mathcal{L}}_{\mathrm{IB},\mathrm{pow}}(T;{\beta }_{\mathrm{pow}},\alpha )=I(T;Y)-{\beta }_{\mathrm{pow}}I{(X;T)}^{(1+\alpha )}$, $\alpha >0$ (Note when $\alpha =1$ we have the squared IB functional from Kolchinsky et al. [21]).

2. the exponential IB Lagrangians: ${\mathcal{L}}_{\mathrm{IB},\exp }(T;{\beta }_{\exp },\eta )=I(T;Y)-{\beta }_{\exp }exp\left(\eta I(X;T)\right)$, $\eta >0$.

Figure 1.

The top row shows the results for the power information bottleneck (IB) Lagrangian with $\alpha =1$, and the bottom row for the exponential IB Lagrangian with $\eta =1$, both in the MNIST dataset. In each row, from left to right it is shown (i) the information plane, where the region of possible solutions of the IB problem is shadowed in light orange and the information-theoretic limits are the dashed orange line; (ii) $I(T;Y)$ as a function of ${\beta }_u$; and (iii) the compression $I(X;T)$ as a function of ${\beta }_u$. In all plots, the red crosses joined by a dotted line represent the values computed with the training set, the blue dots the values computed with the validation set and the green stars the theoretical values computed as dictated by Proposition 3. Moreover, in all plots, it is indicated $I(X;Y)=H\left(Y\right)={log}_2\left(10\right)$ in a dashed, orange line. All values are shown in bits.

We can clearly see how both Lagrangians are able to explore the IB curve (first column from Figure 1) and how the theoretical performance trend of the Lagrangians matches the experimental results (second and third columns from Figure 1). There are small mismatches between the theoretical and experimental performance. This is because using the nonlinear-IB, as stated by Kolchinsky et al. [21], does not guarantee that we find optimal representations due to factors like (i) inaccurate estimation of $I(X;T)$, (ii) restrictions on the structure of T, (iii) use of an estimation of the decoder instead of the real one and (iv) the typical non-convex optimization issues that arise with gradient-based methods. The main difference comes from the discontinuities in performance for increasing $\beta$, which cause is still unknown (cf. Wu et al. [27]). It has been observed, however, that the bottleneck variable performs an intrinsic clusterization in classification tasks (see, for instance, [21,26,42] or Figure 2b). We observed how this clusterization matches with the quantized performance levels observed (e.g., compare Figure 2a with the top center graph in Figure 1); with maximum performance when the number of clusters is equal to the cardinality of Y and reducing performance with a reduction of the number of clusters, which is in line with the concurrent work from Wu and Fischer [43]. We do not have a mathematical proof for the exact relationship between these two phenomena; however, we agree with Wu et al. [27] that it is an interesting matter and hope this observation serves as motivation to derive new theory.

Figure 2.

Depiction of the clusterization behavior of the bottleneck variable for the power IB Lagrangian in the MNIST dataset with $\alpha =1$. The clusters were obtained using the DBSCAN algorithm [44,45].

In practice, there are different criteria for choosing the function u. For instance, the exponential IB Lagrangian could be more desirable than the power IB Lagrangian when we want to draw the IB curve since it has a finite range of ${\beta }_u$. This is $B_u=[{(\eta exp\left(\eta I_{\max }\right))}^{-1},{\eta }^{-1}]$ for the exponential IB Lagrangian vs. $B_u=[{\left((1+\alpha )I_{\max }^{\alpha }\right)}^{-1},\infty )$ for the power IB Lagrangian. Furthermore, there is a trade-off between (i) how much the selected u function resembles a linear function in our region of interest; e.g., with $\alpha$ or $\eta$ close to zero, since it will suffer from similar problems as the original IB Lagrangian; and (ii) how fast it grows in our region of interest; e.g., higher values of $\alpha$ or $\eta$, since it will suffer from value convergence; i.e., optimizing for separate values of ${\beta }_u$ will achieve similar levels of performance (Figure 3). Please, refer to Appendix I for a more thorough explanation of these two phenomena.

Figure 3.

Example of value convergence with the exponential IB Lagrangian with $\eta =3$. We show the intersection of the isolines of ${\mathcal{L}}_{\mathrm{IB},\exp }(T;{\beta }_{\exp })$ for different ${\beta }_{\exp }\in B_{\exp }\approx [1.56\times {10}^{-5},3^{-1}]$ using Corollary 2.

Particularly, the value convergence phenomenon can be exploited in order to approximately obtain a particular level of compression $r^{\ast }$, both for known and unkown IB curves (see Appendix I or the example in Figure 4). For known IB curves, we also know the achieved predictability $I(T;Y)$ since it is the same as the level of compression $I(X;T)$. For this exploitation, we can employ the shifted version of the exponential IB Lagrangian (which is also a particular case of the convex IB Lagrangian):

• the shifted exponential IB Lagrangians:

  $${\mathcal{L}}_{\mathrm{IB},\mathrm{sh}-\exp }(T;{\beta }_{\mathrm{sh}-\exp },\eta ,r^{\ast })=I(T;Y)-{\beta }_{\mathrm{sh}-\exp }exp\left(\eta (I(X;T)-r^{\ast })\right),\eta >0,r^{\ast }\in [0,\infty ).$$

Figure 4.

Example of value convergence exploitation with the shifted exponential Lagrangian with $\eta =200$. In the top row, for the MNIST dataset aiming for a compression level $r^{\ast }=2$ and in the bottom row, for the TREC-6 dataset aiming for a compression level of $r^{\ast }=16$. In each row, from left to right it is shown (i) the information plane, where the region of possible solutions of the IB problem is shadowed in light orange and the information-theoretic limits are the dashed orange line; (ii) $I(T;Y)$ as a function of ${\beta }_u$; and (iii) the compression $I(X;T)$ as a function of ${\beta }_u$. In all plots, the red crosses joined by a dotted line represent the values computed with the training set, the blue dots the values computed with the validation set and the green stars the theoretical values computed as dictated by Proposition 3. Moreover, in all plots, it is indicated $H\left(Y\right)$ in a dashed, orange line. All values are shown in bits.

For this Lagrangian, the optimization procedure converges to representations with approximately the desired compression level $r^{\ast }$ if the hyperparameter $\eta$ is set to a large value.

In Figure 4 we show the results of aiming for a compression level of $r^{\ast }=2$ bits in the MNIST dataset and of $r^{\ast }=16$ bits in the TREC-6 dataset, both with $\eta =200$. We can see how for different values of ${\beta }_{\mathrm{sh}-\exp }$ we can obtain the same desired compression level, which makes this method stable to variations in the Lagrange multiplier selection.

To sum up, in order to achieve a desired level of performance with the convex IB Lagrangian as an objective one should:

1. In a deterministic or close to a deterministic setting (see $ϵ$-deterministic definition in Kolchinsky et al. [21]): Use the adequate ${\beta }_u$ for that performance using Proposition 3. Then if the performance is lower than desired, i.e., we are placed in the wrong performance plateau, gradually reduce the value of ${\beta }_u$ until reaching the previous performance plateau. Alternatively, exploit the value convergence phenomenon with, for instance, the shifted exponential IB Lagrangian.

2. In a stochastic setting: exploit the value convergence phenomenon with, for instance, the shifted exponential IB Lagrangian. Alternatively, draw the IB curve with multiple values of ${\beta }_u$ on the range defined by Corollary 3 and select the representations that best fit their interests.

6. Conclusions

The information bottleneck is a widely used and studied technique. However, it is known that the IB Lagrangian cannot be used to achieve varying levels of performance in deterministic scenarios. Moreover, in order to achieve a particular level of performance, multiple optimizations with different Lagrange multipliers must be done to draw the IB curve and select the best traded-off representation.

In this article we introduced a general family of Lagrangians which allow to (i) achieve varying levels of performance in any scenario, and (ii) pinpoint a specific Lagrange multiplier ${\beta }_u$ to optimize for a specific performance level in known IB curve scenarios; e.g., deterministic. Furthermore, we showed the ${\beta }_u$ domain when the IB curve is known and a ${\beta }_u$ domain bound for exploring the IB curve when it is unknown. This way we can reduce and/or avoid multiple optimizations and, hence, reduce the computational effort for finding well traded-off representations. Moreover, (iii) when the IB curve is not known, we saw how we can exploit the value convergence issue of the convex IB Lagrangian to approximately obtain a specific compression level for both known and unknown IB curve shapes. Finally, (iv) we provided some insight into the discontinuities on the performance levels w.r.t. the Lagrange multipliers by connecting those with the intrinsic clusterization of the bottleneck variable.

Appendix A. Proof of Proposition 1

Proof. 

We can easily prove this statement by finding $I(T;Y)$ is lower bounded by the $\gamma J_{\mathrm{CE}}(p_{(X,Y)};\theta )+C$ where $\gamma <0$ and C does not depend on T. This way maximizing such lower bound would be equivalent to minimizing $J_{\mathrm{CE}}(p_{(X,Y)};\theta )$ and, moreover, it would imply maximizing $I(T;Y)$.

We can find such an expression as follows:

$$\begin{array}{cc}\hfill I(T;Y)& ={\mathbb{E}}_{(y,t)\sim q_{Y|T}{q}_T}\left[log\left(\frac{q_{Y|T=t}(y|t;\theta )}{p_Y\left(y\right)}\right)\right]=H\left(Y\right)+{\mathbb{E}}_{(y,t)\sim q_{Y|T}{q}_T}\left[log\left(q_{Y|T=t}(y|t;\theta )\right)\right]\hfill \end{array}$$ (A1)

$$\begin{array}{cc}\hfill & =H\left(Y\right)+{\mathbb{E}}_{t\sim q_T}\left[D_{\mathrm{KL}}\left(q_{Y|T=t}||q_{\widehat{Y}|T=t}\right)\right]+{\mathbb{E}}_{(y,t)\sim q_{Y|T}{q}_T}\left[log(q_{\widehat{Y}|T}(y|t;\theta ))\right]\hfill \end{array}$$ (A2)

$$\begin{array}{cc}\hfill & \ge H\left(Y\right)+{\mathbb{E}}_{(x,y,t)\sim q_{Y|T}{q}_{T|X}{p}_X}\left[log(q_{\widehat{Y}|T=t}(y|t,\theta ))\right]=H\left(Y\right)-{\mathbb{E}}_{(x,t)\sim q_{T|X}{p}_X}\left[\mathbb{C}(q_{Y|T=t}||q_{\widehat{Y}|T=t})\right]\hfill \end{array}$$ (A3)

$$\begin{array}{cc}\hfill & =H\left(Y\right)-J_{\mathrm{CE}}(p_{(X,Y)};\theta ).\hfill \end{array}$$ (A4)

Here, in Equation (A1) we just used the definition of the mutual information between two random variables, and then we decoupled it using the definition of the entropy of a variable (Note we used $H(·)$ which is usually employed for discrete variables. However, in this setting $H(·)$ could also refer to the differential entropy $h(·)$ of a continuous random variable since we employed the general definition using the expectation). Then, in Equation (A2) we only multiplied and divided by $q_{\widehat{Y}|T}$ inside the logarithm and employed the definition of the Kullback–Leibler divergence. Finally, in Equation (A3) we first used the fact the Kullback–Leibler divergence is always positive (Theorem 2.6.3 from Cover and Thomas [20]) and then the properties of the Markov chain $T\leftrightarrow X\leftrightarrow Y$.

Therefore, since $H\left(Y\right)$ does not depend on T and we have a negative multiplicative term on $J_{\mathrm{CE}}(p_{(X,Y)};\theta )$ the proposition is proved. □

Appendix B. Alternative Proof of Theorem 1

Proof. 

We will proof all the enumerated statements sequentially, since the third one requires from the two first ones to be proved.

1. Proposition 2 states that the IB curve in the information plane follows the equation $I(T;Y)=I(X;T)$ if $I(X;T)\in [0,I(X;Y\left)\right)$. Then, since $\beta =dI(T;Y)/dI(X;T)$ [1], we know $\beta =1$ in all these points. Therefore, for $\beta =1$ all points ($I(X;T),I(X;T)$) such that $I(X;T)\in [0,I(X;Y\left)\right)$ are solutions of optimizing the IB Lagrangian.

2. Similarly, Proposition 2 states that the IB curve follows the equation $I(T;Y)=I(X;Y)$ if $I(X;T)\ge I(X;Y)$. Then, since $\beta =dI(T;Y)/dI(X;T)$ [1], we know $\beta =0$ in all points such that $I(X;T)>I(X;Y)$. We cannot ensure it at $I(X;T)=I(X;Y)$ since $\beta =1$ for $I(X;T)={lim}_{ϵ\to 0^{+}}\{I(X;Y)-ϵ\}$.

3. Finally, in order to prove the last statement we will first prove that if $\beta \in (0,1)$ achieves a solution, it is $\left(I\right(X;Y),I(X;Y\left)\right)$. Then, we will prove that if the solution $\left(I\right(X;Y),I(X;Y\left)\right)$ exists, this can be yield by any $\beta \in (0,1)$. Hence, the solution $\left(I\right(X;Y),I(X;Y\left)\right)$ is achieved $\forall \beta \in (0,1)$ and it is the only solution achievable.

   • (a)
     Since the IB curve is concave we know $\beta$ is non-increasing in $I(X;T)\in {\mathbb{R}}^{+}$. We also know $\beta =1$ at the points in the IB curve where $I(X;T)\le {lim}_{ϵ\to 0^{+}}\{I(X;Y)-ϵ\}$ and $\beta =1$ at the points in the IB curve where $I(X;T)\ge {lim}_{ϵ\to 0^{+}}\{I(X;Y)+ϵ\}$. Hence, if we achieve a solution with $\beta \in (0,1)$, this solution is $I(X;T)=I(T;Y)=I(X;Y)$.
   • (b)
     We can upper bound the IB Lagrangian by

     $${\mathcal{L}}_{\mathrm{IB}}(T;\beta )=I(T;Y)-\beta I(X;T)\le (1-\beta )I(T;Y)\le (1-\beta )I(X;Y),$$ (A5)

     where the first and second inequalities use the DPI (Theorem 2.8.1 from Cover and Thomas [20]).
     Then, we can consider the point of the IB curve $\left(I\right(X;Y),I(X;Y\left)\right)$. Since the function is concave a tangent line to $\left(I\right(X;Y),I(X;Y\left)\right)$ exists such that all other points in the curve lie below this line. Let $\beta$ be the slope of this curve (which we know it is from Tishby et al. [1]). Then,

     $$I(X;Y)-\beta I(X;Y)=(1-\beta )I(X;Y)\ge F_{\mathrm{IB},\max }\left(r\right)-\beta r,\forall r\in [0,\infty ).$$ (A6)
     As we see, by the upper bound on the IB Lagrangian from Equation (A5), if the point $\left(I\right(X;Y),I(X;Y\left)\right)$ exists, any $\beta$ can be the slope of the tangent line to $\left(I\right(X;Y),I(X;Y\left)\right)$ that ensures concavity. □

Appendix C. Proof of Theorem 2

Proof. 

We start the proof by remembering the optimization problem at hand (Definition 1):

$$F_{\mathrm{IB},\max }\left(r\right)=\underset{T\in \Delta }{max}\left\{I(T;Y)\right\}\mathrm{s}.\mathrm{t}.I(X;T)\le r$$ (A7)

We can modify the optimization problem by

$$\begin{array}{c}\hfill \underset{T\in \Delta }{max}\left\{I(T;Y)\right\}\mathrm{s}.\mathrm{t}.u\left(I(X;T)\right)\le u\left(r\right)\end{array}$$ (A8)

iff u is a monotonically non-decreasing function since otherwise $u\left(I\right(X;T\left)\right)\le u\left(r\right)$ would not hold necessarily. Now, let us assume $\exists T^{\ast }\in \Delta$ and ${\beta }_u^{\ast }$ s.t. $T^{\ast }$ maximizes ${\mathcal{L}}_{\mathrm{IB},u}(T;{\beta }_u^{\ast })$ over all $T\in \Delta$, and $I(X;T^{\ast })\le r$. Then, we can operate as follows:

$$\begin{array}{cc}\hfill \underset{\begin{array}{c}T\in \Delta \\ u\left(I\right(X;T\left)\right)\le u\left(r\right)\end{array}}{max}\left\{I(T;Y)\right\}& =\underset{\begin{array}{c}T\in \Delta \\ u\left(I\right(X;T\left)\right)\le u\left(r\right)\end{array}}{max}\{I(T;Y)-{\beta }_u^{\ast }(u\left(I(X;T)\right)-u\left(r\right)+\xi )\}\hfill \end{array}$$ (A9)

$$\begin{array}{cc}\hfill & \le \underset{T\in \Delta }{max}\{I(T;Y)-{\beta }_u^{\ast }(u\left(I(X;T)\right)-u\left(r\right)+\xi )\}\hfill \end{array}$$ (A10)

$$\begin{array}{cc}\hfill & =I(T^{\ast };Y)-{\beta }_u^{\ast }(u(I(X;T^{\ast })-u\left(r\right)+\xi )=I(T^{\ast };Y).\hfill \end{array}$$ (A11)

Here, the equality from Equation (A9) comes from the fact that since $I(X;T)\le r$, then $\exists \xi \ge 0$ s.t. $u\left(I\right(X;T\left)\right)-u\left(r\right)+\xi =0$. Then, the inequality from Equation (A10) holds since we have expanded the optimization search space. Finally, in Equation (A11) we use that $T^{\ast }$ maximizes ${\mathcal{L}}_{\mathrm{IB},u}(T;{\beta }_u^{\ast })$ and that $I(X;T^{\ast })\le r$.

Now, we can exploit that $u\left(r\right)$ and $\xi$ do not depend on T and drop them in the maximization in Equation (A10). We can then realize we are maximizing over ${\mathcal{L}}_{\mathrm{IB},u}(T;{\beta }_u^{\ast })$; i.e.,

$$\begin{array}{cc}\hfill \underset{\begin{array}{c}T\in \Delta \\ u\left(I\right(X;T\left)\right)\le u\left(r\right)\end{array}}{max}\left\{I(T;Y)\right\}& \le \underset{T\in \Delta }{max}\{I(T;Y)-{\beta }_u^{\ast }(u\left(I(X;T)\right)-u\left(r\right)+\xi )\}\hfill \end{array}$$ (A12)

$$\begin{array}{cc}\hfill & =\underset{T\in \Delta }{max}\{I(T;Y)-{\beta }_u^{\ast }\left(I(X;T)\right)\}=\underset{T\in \Delta }{max}\left\{{\mathcal{L}}_{\mathrm{IB},u}(T;{\beta }_u^{\ast })\right\}.\hfill \end{array}$$ (A13)

Therefore, since $I(T^{\ast };Y)$ satisfies both the maximization with $T^{\ast }\in \Delta$ and the constraint $I(X;T^{\ast })\le r$, maximizing ${\mathcal{L}}_{\mathrm{IB},u}(T;{\beta }_u^{\ast })$ obtains $F_{\mathrm{IB},\max }\left(r\right)$.

Now, we know if such ${\beta }_u^{\ast }$ exists, then the solution of the Lagrangian will be a solution for $F_{\mathrm{IB},\max }\left(r\right)$. Then, if we consider Theorem 6 from the Appendix of Courcoubetis [22] and consider the maximization problem instead of the minimization problem, we know if both $I(T;Y)$ and $-u\left(I\right(X;T\left)\right)$ are concave functions, then a set of Lagrange multipliers $S_u^{\ast }$ exists with these conditions. We can make this consideration because f is concave if $-f$ is convex and $max\left\{f\right\}=min\{-f\}$. We know $I(T;Y)$ is a concave function of T for $T\in \Delta$ (Lemma 5 of Gilad-Bachrach et al. [19]) and $I(X;T)$ is convex w.r.t. T given $p_X$ is fixed (Theorem 2.7.4 of Cover and Thomas [20]). Thus, if we want $-u\left(I\right(X;T\left)\right)$ to be concave we need u to be a convex function.

Finally, we will look at the conditions of u so that for every point $\left(I\right(X;T),I(T;Y\left)\right)$ in the IB curve, there exists a unique ${\beta }_u^{\ast }$ s.t. ${\mathcal{L}}_{\mathrm{IB},u}(T;{\beta }_u^{\ast })$ is maximized. That is, the conditions of u s.t. $|S_u^{\ast }|=1$. For this purpose we will look at the solutions of the Lagrangian optimization:

$$\frac{d{\mathcal{L}}_{\mathrm{IB},u}(T;{\beta }_u)}{dT}=\frac{d(I(T;Y)-{\beta }_{u}u\left(I(X;T)\right))}{dT}=\frac{dI(T;Y)}{dT}-{\beta }_u\frac{du\left(I\right(X;T\left)\right)}{dI(X;T)}\frac{dI(X;T)}{dT}=0$$ (A14)

Now, if we integrate both sides of Equation (A14) over all $T\in \Delta$ we obtain

$${\beta }_u=\frac{dI(T;Y)}{dI(X;T)}{\left(\frac{du\left(I\right(X;T\left)\right)}{dI(X;T)}\right)}^{-1}=\frac{\beta }{u^{\prime }\left(I(X;T)\right)},$$ (A15)

where $\beta$ is the Lagrange multiplier from the IB Lagrangian [1] and $u^{\prime }\left(I(X;T)\right)$ is $\frac{du\left(I\right(X;T\left)\right)}{dI(X;T)}$. Also, if we want to avoid indeterminations of ${\beta }_u$ we need $u^{\prime }\left(I(X;T)\right)$ not to be 0. Since we already imposed u to be monotonically non-decreasing, we can solve this issue by strengthening this condition. That is, we will require u to be monotonically increasing.

We would like ${\beta }_u$ to be continuous, this way there would be a unique ${\beta }_u$ for each value of $I(X;T)$. We know $\beta$ is a non-increasing function of $I(X;T)$ (Lemma 6 of Gilad-Bachrach et al. [19]). Hence, if we want ${\beta }_u$ to be a strictly decreasing function of $I(X;T)$, we will require $u^{\prime }$ to be a strictly increasing function of $I(X;T)$. Therefore, we will require u to be a strictly convex function.

Thus, if u is a strictly convex and monotonically increasing function, for each point $\left(I\right(X;T),I(T;Y\left)\right)$ in the IB curve s.t. $dI(T;Y)/dI(X;T)>0$ there is a unique ${\beta }_u$ for which maximizing ${\mathcal{L}}_{\mathrm{IB},u}(T;{\beta }_u)$ achieves this solution. □

Appendix D. Proof of Proposition 3

Proof. 

In Theorem 2 we showed how each point of the IB curve $\left(I\right(X;T),I(T;Y\left)\right)$ can be found with a unique ${\beta }_u$ maximizing ${\mathcal{L}}_{\mathrm{IB},u}(T;{\beta }_u)$. Therefore, since we also proved ${\mathcal{L}}_{\mathrm{IB},u}(T;{\beta }_u)$ is strictly concave w.r.t. T we can find the values of ${\beta }_u$ that maximize the Lagrangian for fixed $I(X;T)$.

First, we look at the solutions of the Lagrangian maximization:

$$\frac{d{\mathcal{L}}_{\mathrm{IB},u}(T;{\beta }_u)}{dT}=\frac{d(f_{\mathrm{IB}}\left(I(X;T)\right)-{\beta }_{u}u\left(I(X;T)\right))}{dT}=\frac{d{f}_{\mathrm{IB}}\left(I(X;T)\right)}{dT}-{\beta }_u\frac{du\left(I\right(X;T\left)\right)}{dI(X;T)}\frac{dI(X;T)}{dT}=0.$$ (A16)

Then as before we can integrate at both sides for all $T\in \Delta$ and solve for ${\beta }_u$:

$${\beta }_u=\frac{d{f}_{\mathrm{IB}}\left(I(X;T)\right)}{dI(X;T)}\frac{1}{u^{\prime }\left(I(X;T)\right)}.$$ (A17)

Moreover, since u is a strictly convex function it’s derivative $u^{\prime }$ is strictly increasing. Hence, $u^{\prime }$ is an invertible function (since a strictly increasing function is bijective and a function is invertible iff it is bijective by definition). Now, if we consider ${\beta }_u>0$ to be known and $I(X;T)$ to be the unknown we can solve for $I(X;T)$ and get:

$$I(X;T)={\left(u^{\prime }\right)}^{-1}\left(\frac{d{f}_{\mathrm{IB}}\left(I(X;T)\right)}{dI(X;T)}\frac{1}{{\beta }_u}\right).$$ (A18)

Note we require ${\beta }_u$ not to be 0 so the mapping is defined. □

Appendix E. Proof of Corollary 2

Proof. 

We will start the proof by proving the following useful Lemma.

Lemma A1.

Let ${\mathcal{L}}_{\mathrm{IB},u}(T;{\beta }_u)$ be a convex IB Lagrangian, then ${sup}_{T\in \Delta }\left\{{\mathcal{L}}_{\mathrm{IB},u}(T;0)\right\}=I(X;Y)$.

Proof. 

Since ${\mathcal{L}}_{\mathrm{IB},u}(T;0)=I(T;Y)$, maximizing this Lagrangian is directly maximizing $I(T;Y)$. We know $I(T;Y)$ is a concave function of T for $T\in \Delta$ (Theorem 2.7.4 from Cover and Thomas [20]); hence it has a supremum. We also know $I(T;Y)\le I(X;Y)$. Moreover, we know $I(X;Y)$ can be achieved if, for example, Y is a deterministic function of T (since then the Markov Chain $X\leftrightarrow T\leftrightarrow Y$ is formed). Thus, ${sup}_{T\in \Delta }\left\{{\mathcal{L}}_{\mathrm{IB},u}(T;0)\right\}=I(X;Y)$. □

For ${\beta }_u=0$ we know maximizing ${\mathcal{L}}_{\mathrm{IB},u}(T;0)$ we can obtain the point in the IB curve $(r_{\max },I_{\max })$ (Lemma A1). Moreover, we know that for every point $(I(X;T),f_{\mathrm{IB}}\left(I(X;T)\right))$ such that $d{f}_{\mathrm{IB}}\left(I(X;T)\right)/dI(X;T)>0$, $\exists !{\beta }_u$ s.t. $max\left\{{\mathcal{L}}_{\mathrm{IB},u}(T;{\beta }_u)\right\}$ achieves that point (Theorem 2). Thus, $\exists !{\beta }_{u,\min }$ s.t. ${lim}_{r\to r_{\max }^{-}}(r,f_{\mathrm{IB}}\left(r\right))$ is achieved. From Proposition 3 we know this ${\beta }_{u,\min }$ is given by

$${\beta }_{u,\min }=\underset{r\to r_{\max }^{-}}{lim}\left\{\frac{f_{\mathrm{IB}}^{\prime }\left(r\right)}{u^{\prime }\left(r\right)}\right\}.$$ (A19)

Since we know $f_{\mathrm{IB}}\left(I(X;T)\right)$ is a concave non-decreasing function in $(0,r_{\max })$ (Lemma 5 of Gilad-Bachrach et al. [19]) we know it is continuous in this interval. In addition we know ${\beta }_u$ is strictly decreasing w.r.t. $I(X;T)$ (Theorem 2). Furthermore, by definition of $r_{\max }$ and knowing $I(T;Y)\le I(X;Y)$ we know $f_{\mathrm{IB}}^{\prime }\left(r\right)=0$, $\forall r>r_{\max }$. Therefore, we cannot ensure the exploration of the IB curve for ${\beta }_u^{\prime }$ s.t. $0<{\beta }_u^{\prime }<{\beta }_{u,\min }$.

Then, since u is a strictly increasing function in $(0,r_{\max })$, $u^{\prime }$ is positive in that interval. Hence, taking into account ${\beta }_u$ is strictly decreasing we can find a maximum ${\beta }_u$ when $I(X;T)$ approaches to 0. That is,

$${\beta }_{u,\max }=\underset{r\to 0^{+}}{lim}\left\{\frac{f_{\mathrm{IB}}^{\prime }\left(r\right)}{u^{\prime }\left(r\right)}\right\},$$ (A20)

 □

Appendix F. Proof of Corollary 3

Proof. 

If we use Corollary 2, it is straightforward to see that ${\beta }_u\subseteq [L_{-},L_{+}]$ if ${\beta }_{u,\min }\ge L_{-}$ and ${\beta }_{u,\max }\le L_{+}$ for all IB curves $f_{\mathrm{IB}}$ and functions u. Therefore, we look at a domain bound dependent on the function choice. That is, if we can find ${\beta }_{\min }\le f_{\mathrm{IB}}^{\prime }\left(r\right)$ and ${\beta }_{\max }\ge f_{\mathrm{IB}}^{\prime }\left(r\right)$ for all IB curves and all values of r, then

$$B_u\subseteq \left[\frac{{\beta }_{\min }}{{lim}_{r\to r_{\max }^{-}}\left\{u^{\prime }\left(r\right)\right\}},\frac{{\beta }_{\max }}{{lim}_{r\to 0^{+}}\left\{u^{\prime }\left(r\right)\right\}}\right].$$ (A21)

The region for all possible IB curves regardless of the relationship between X and Y is depicted in Figure A1. The hard limits are imposed by the DPI (Theorem 2.8.1 from Cover and Thomas [20]) and the fact that the mutual information is non-negative (Corollary with Equation 2.90 for discrete and first Corollary of Theorem 8.6.1 for continuous random variables from Cover and Thomas [20]). Hence, a minimum and maximum values of $f_{\mathrm{IB}}^{\prime }$ are given by the minimum and maximum values of the slope of the Pareto frontier. Which means

$$B_u\subseteq \left[0,\frac{1}{{lim}_{r\to 0^{+}}\left\{u^{\prime }\left(r\right)\right\}}\right].$$ (A22)

Note $0/\left({lim}_{r\to r_{max}^{-}}\left\{u^{\prime }\left(r\right)\right\}\right)=0$ since u is monotonically increasing and, thus, $u^{\prime }$ will never be 0.

Then, we can tighten the bound using the results from Wu et al. [27], where, in Theorem 2, they showed the slope of the Pareto frontier could be bounded in the origin by $f_{\mathrm{IB}}^{\prime }\le {\left({inf}_{{\Omega }_x\subset \mathcal{X}}\left\{{\beta }_0\left({\Omega }_x\right)\right\}\right)}^{-1}$. Finally, we know that in deterministic classification tasks ${inf}_{{\Omega }_x\subset \mathcal{X}}\left\{{\beta }_0\left({\Omega }_x\right)\right\}=1$, which aligns with Kolchinsky et al. [21] and what we can observe from Figure A1. Therefore,

$$B_u\subseteq \left[0,\frac{{\left({inf}_{{\Omega }_x\subset \mathcal{X}}\left\{{\beta }_0\left({\Omega }_x\right)\right\}\right)}^{-1}}{{lim}_{r\to 0^{+}}\left\{u^{\prime }\left(r\right)\right\}}\right]\subseteq \left[0,\frac{1}{{lim}_{r\to 0^{+}}\left\{u^{\prime }\left(r\right)\right\}}\right].$$ (A23)

 □

Figure A1.

Graphical representation of the IB curve in the information plane. Dashed lines in orange represent tight bounds confining the region (in light orange) of possible IB curves (delimited by the red line, also known as the Pareto frontier). Black dotted lines are informative values. In blue we show an example of a possible IB curve confining a region (in darker orange) of an IB curve that does not achieve the Pareto frontier. Finally, the yellow star represents the point where the representation keeps the same information about the input and the output.

Appendix G. Other Lagrangian Families

We can use the same ideas we used for the convex IB Lagrangian to formulate new families of Lagrangians that allow the exploration of the IB curve. For that, we will use the duality of the IB curve (Lemma 10 of [19]). That is:

Definition A1 (IB Dual Functional).

Let X and Y be statistically dependent variables. Let also Δ be the set of random variables T obeying the Markov condition $Y\leftrightarrow X\leftrightarrow T$. Then the IB dual functional is

$$F_{\mathrm{IB},\min }\left(i\right)=\underset{T\in \Delta }{min}\left\{I(X;T)\right\}\mathrm{s}.\mathrm{t}.I(T;Y)\ge i,\forall i\in [0,I(X;Y)).$$ (A24)

Theorem A1 (IB Curve Duality).

Let the IB curve be defined by the solutions of $F_{\mathrm{IB},\max }\left(r\right)$ for varying $r\in [0,\infty )$. Then,

$$\forall r\exists is.t.(r,F_{\mathrm{IB},\max }\left(r\right))=(F_{\mathrm{IB},\min }\left(i\right),i)$$ (A25)

and

$$\forall i\exists rs.t.(F_{\mathrm{IB},\min }\left(i\right),i)=(r,F_{\mathrm{IB},\max }\left(r\right)).$$ (A26)

From this definition, it follows that minimizing the dual IB Lagrangian, ${\mathcal{L}}_{\mathrm{IB},\mathrm{dual}}(T;{\beta }_{\mathrm{dual}})=I(X;T)-{\beta }_{\mathrm{dual}}I(T;Y)$, for ${\beta }_{\mathrm{dual}}={\beta }^{-1}$ is equivalent to maximizing the IB Lagrangian. In fact, the original Lagrangian for solving the problem was defined this way [1]. We decided to use the maximization version because the domain of useful $\beta$ is bounded while it is not for ${\beta }_{\mathrm{dual}}$.

Following the same reasoning as we did in the proof of Theorem 2, we can ensure the IB curve can be explored if:

1. We minimize the concave IB Lagrangian ${\mathcal{L}}_{\mathrm{IB},v}(T;{\beta }_v)=I(X;T)-{\beta }_{v}v\left(I(T;Y)\right)$.

2. We maximize the dual concave IB Lagrangian ${\mathcal{L}}_{\mathrm{IB},v,\mathrm{dual}}(T;{\beta }_{v,\mathrm{dual}})=v\left(I(T;Y)\right)-{\beta }_{v,\mathrm{dual}}I(X;T)$.

3. We minimize the dual convex IB Lagrangian ${\mathcal{L}}_{\mathrm{IB},u,\mathrm{dual}}(T;{\beta }_{u,\mathrm{dual}})=u\left(I(X;T)\right)-{\beta }_{u,\mathrm{dual}}I(T;Y)$.

Here, u is a monotonically increasing strictly convex function, v is a monotonically increasing strictly concave function, and ${\beta }_v,{\beta }_{v,\mathrm{dual}},{\beta }_{u,\mathrm{dual}}$ are the Lagrange multipliers of the families of Lagrangians defined above.

In a similar manner, one could obtain relationships between the Lagrange multipliers of the IB Lagrangian and the convex IB Lagrangian with these Lagrangian families. For instance, the convex IB Lagrangian ${\mathcal{L}}_{\mathrm{IB},u}(T;{\beta }_u)$ is related with the concave IB Lagrangian ${\mathcal{L}}_{\mathrm{IB},v}(T;{\beta }_v)$ as defined by Propositon A1.

Proposition A1 (Relationship between the convex and concave IB Lagrangians).

Consider the convex and concave IB Lagrangians ${\mathcal{L}}_{\mathrm{IB},u}(T;{\beta }_u)$, ${\mathcal{L}}_{\mathrm{IB},v}(T;{\beta }_v)$. Let the IB curve defined as in Definition 2 be $f_{\mathrm{IB}}$. Then, if we fix the functions u and v we can obtain the same point in the IB curve $(r,f_{\mathrm{IB}}\left(r\right))$ with both Lagrangians when

$${\beta }_v^{-1}=f_{\mathrm{IB}}^{\prime }\left(r\right)v^{\prime }\left(f_{\mathrm{IB}}\left({\left(u^{\prime }\right)}^{-1}\left(\frac{f_{\mathrm{IB}}^{\prime }\left(r\right)}{{\beta }_u}\right)\right)\right),$$ (A27)

or equivalently,

$${\beta }_u^{-1}=\frac{1}{f_{\mathrm{IB}}^{\prime }\left(r\right)}{u}^{\prime }\left(f_{\mathrm{IB}}^{-1}\left({\left(v^{\prime }\right)}^{-1}\left(\frac{{\beta }_v^{-1}}{f_{\mathrm{IB}}^{\prime }\left(r\right)}\right)\right)\right).$$ (A28)

Proof. 

If we proceed like we did in the proof of Proposition 3 we can find the mapping between $I(X;T)$ and ${\beta }_u$ and between $I(T;Y)$ and ${\beta }_v$. That is,

$$I(X;T)={\left(u^{\prime }\right)}^{-1}\left(\frac{d{f}_{\mathrm{IB}}\left(I(X;T)\right)}{dI(X;T)}\frac{1}{{\beta }_u}\right)\mathrm{and}I(T;Y)={\left(v^{\prime }\right)}^{-1}\left({\left(\frac{d{f}_{\mathrm{IB}}\left(I(X;T)\right)}{dI(X;T)}\right)}^{-1}\frac{1}{{\beta }_v}\right).$$ (A29)

Then, if we recall that $I(T;Y)=f_{\mathrm{IB}}\left(I(X;T)\right)$, we can directly obtain that

$$f_{\mathrm{IB}}\left({\left(u^{\prime }\right)}^{-1}\left(\frac{d{f}_{\mathrm{IB}}\left(I(X;T)\right)}{dI(X;T)}\frac{1}{{\beta }_u}\right)\right)={\left(v^{\prime }\right)}^{-1}\left({\left(\frac{d{f}_{\mathrm{IB}}\left(I(X;T)\right)}{dI(X;T)}\right)}^{-1}\frac{1}{{\beta }_v}\right).$$ (A30)

Then, if we solve Equation (A30) with a fixed point $(I(X;T)=r,I(T;Y)=f_{\mathrm{IB}}\left(r\right))$ for ${\beta }_v$ we obtain Equation (A27), and if we solve it for ${\beta }_u$ we obtain Equation (A28). □

Also, one could find a range of values for these Lagrangians to allow for the IB curve exploration and define a bijective mapping between their Lagrange multipliers and the IB curve. However, (i) as mentioned in Section 2.2, $I(T;Y)$ is particularly interesting to maximize without transformations because of its meaning. Moreover, (ii) like ${\beta }_{\mathrm{dual}}$, the domain of useful ${\beta }_v$ and ${\beta }_{u,\mathrm{dual}}$ is not upper bounded. These two reasons make these other Lagrangians less preferable. We only include them here for completeness. Nonetheless, we encourage the curiours reader to explore these families of Lagrangians too. For example, a possible interesting research would be investigating if some particularization of the concave IB Lagrangian suffers from an issue like value convergence that can be exploited for approximately obtaining any predictability level $I(T;Y)=i^{\ast }$ for many values of ${\beta }_v$.

Appendix H. Experimental Setup Details and Further Experiments

In order to generate empirical support for our claims, we performed several experiments on different datasets with different neural network architectures and different ways of calculating the information bottleneck.

Appendix H.1. Information Bottleneck Calculations

The information bottleneck is calculated modifying either the nonlinear-IB [26]. This method of calculating the information bottleneck is a neural network that minimizes the cross-entropy while also miniminizing an upper bound estimate of the mutual information $I_{\theta }\approx I(X;T)$. The nonlinear-IB relies on a kernel-based estimate of this mutual information [40]. We modify this calculation method by applying the function u to the $I(X;T)$ estimate.

For the nonlinear-IB calculations, we estimated the gradients of both $I_{\theta }(X;T)$ and the cross-entropy with the same mini-batch. Moreover, we did not learn the covariance of the mixture of Gaussians used for the kernel density estimation of $I_{\theta }(X;T)$ and we set it to ${(exp(-1))}^2$.

In both methods, and for all the experiments, we assumed a Gaussian stochastic encoder $T=f_{\mathrm{enc}}(X;\theta )+W$ with $p_W=\mathcal{N}(0,I_d)$, where d are the number of dimensions of the representations. We trained the neural networks with the Adam optimization algorithm [46] with a learning rate of ${10}^{-4}$ and a $0.6$ decay rate every 10 epochs. We used a batch size of 128 samples and all the weights were initialized according to the method described by Glorot and Bengio [47] using a Gaussian distribution.

Then, we used the DBSCAN algorithm [44,45] for clustering. Particularly, we used the scikit-learn [48] implementation with $ϵ=0.3$ and min_samples = 50.

The reader can find the PyTorch [30] implementation in the following link: https://github.com/burklight/convex-IB-Lagrangian-PyTorch.

Appendix H.2. The Experiments

We performed experiments in four different datasets:

• A Classification Task on the MNIST Dataset [28] (Figure 1, Figure 2, Figure A2, Figure A3 and Figure A4 and top row from Figure 3). This dataset contains 60,000 training samples and 10,000 testing samples of hand-written digits. The samples are 28x28 pixels and are labeled from 0 to 9; i.e., $\mathcal{X}={\mathbb{R}}^{784}$ and $\mathcal{Y}=\{0,1,\dots ,9\}$. The data is pre-processed so that the input has zero mean and unit variance. This is a deterministic setting, hence the experiment is designed to showcase how the convex IB Lagrangians allow us to explore the IB curve in a setting where the normal IB Lagrangian cannot and the relationship between the performance plateaus and the clusterization phenomena. Furthermore, it intends to showcase the behavior of the power and exponential Lagrangians with different parameters of $\alpha$ and $\eta$. Finally, it wants to demonstrate how the value convergence can be employed to approximately obtain a specific compression value. In this experiment, the encoder $f_{\mathrm{enc}}$ is a three fully-connected layer encoder with 800 ReLU units on the first two layers and two linear units on the last layer ($T\in {\mathbb{R}}^2$), and the decoder $f_{\mathrm{dec}}$ is a fully-connected 800 ReLU unit layers followed by an output layer with 10 softmax units. The convex IB Lagrangian was calculated using the nonlinear-IB.

  In Figure A2 we show how the IB curve can be explored with different values of $\alpha$ for the power IB Lagrangian and in Figure A3 for different values of $\eta$ and the exponential IB Lagrangian.

  Finally, in Figure A4 we show the clusterization for the same values of $\alpha$ and $\eta$ as in Figure A2 and Figure A3. In this way the connection between the performance discontinuities and the clusterization is more evident. Furthermore, we can also observe how the exponential IB Lagrangian maintains better the theoretical performance than the power IB Lagrangian (see Appendix I for an explanation of why).

• A Classification Task on the Fashion-MNIST Dataset [49] (Figure A5). As MNSIT, this dataset contains 60,000 training and 10,000 testing samples of 28x28 pixel images labeled from 0 to 9 and constitutes a deterministic setting. The difference is that this dataset contains fashion products instead of hand-written digits and it represents a harder classification task [49]. The data is also pre-processed so that the input has zero mean and unit variance. For this experiment, the encoder $f_{\mathrm{enc}}$ is composed of a two-layer convolutional neural network (CNN) with 32 filters on the first layer and 128 filters on the second with kernels of size 5 and stride 2. This CNN is followed by two fully-connected layers of 128 linear units ($T\in {\mathbb{R}}^{128}$). After the first convolution and the first fully-connected layer, a ReLU activation is employed. The decoder $f_{\mathrm{dec}}$ is a fully-connected 128 ReLU unit layer followed by an output layer with 10 softmax units. The convex IB Lagrangian was calculated using the nonlinear-IB. Therefore, this experiment intends to showcase how the convex IB Lagrangian can explore the IB curve for different neural network architectures and harder datasets.

• A Regression Task on the California Housing Dataset [50] (Figure A6). This dataset contains 20,640 samples of 8 real number input variables like the longitude and latitude of the house (i.e., $X\in {\mathbb{R}}^8$) and a task output real variable representing the price of the house (i.e., $Y\in \mathbb{R}$). We used the log-transformed house price as the target variable and dropped the 992 samples in which the house price was equal or greater than $$\mathrm{500,000}$ so that the output distribution was closer to a Gaussian as they did in [26]. The input variables were processed so that they had zero mean and unit variance and we randomly split the samples into a 70% training and 30% test dataset. As in [40], for regression tasks we approximate $H\left(Y\right)$ with the entropy of a Gaussian with variance $\mathrm{Var}\left(Y\right)$ and $H(Y|T)$ with the entropy of a Gaussian with variance equal to the mean-squared error (MSE). This leads to the estimate $I(T;Y)\approx 0.5log\left(\mathrm{Var}\right(Y)/MSE)$. The encoder $f_{\mathrm{enc}}$ is a three fully-connected layer encoder with 128 ReLU units on the first two layers and 2 linear units on the last layer ($T\in {\mathbb{R}}^2$), and the decoder $f_{\mathrm{dec}}$ is a fully-connected 128 ReLU unit layers followed by an output layer with 1 linear unit. The convex IB Lagrangian was calculated using the nonlinear-IB. Hence, this experiment was designed to showcase the convex IB Lagrangian can explore the IB curve in stochastic scenarios for regression tasks.

• A Classification Task on the TREC-6 Dataset [29] (Figure A7 and bottom row from Figure 3). This dataset is the six-class version of the TREC [51] dataset. It contains 5452 training and 500 test samples of text questions. Each question is labeled within six different semantic categories based on what the answer is; namely: Abbreviation, description and abstract concepts, entities, human beings, locations, and numeric values. This dataset does not constitute a deterministic setting since there are examples that could belong to more than one class and there are examples which are wrongly labeled (e.g., “What is a fear of parasites?” could belong both to the description and abstract concept category, however it is labeled into the entity category), and hence $H(Y|X)>0$. Following Ben Trevett’s tutorial on Sentiment Analysis [52] the encoder $f_{\mathrm{enc}}$ is composed by a 6 billion token pre-trained 100-dimensional Glove word embedding [53], followed by a concatenation of three convolutions with kernel sizes 2–4 respectively, and finalized with a fully-connected 128 linear unit layer ($T\in {\mathbb{R}}^{128}$). The decoder $f_{\mathrm{dec}}$ is a single fully-connected 6 softmax unit layer. The convex IB Lagrangian was calculated using the nonlinear-IB. Thus, this experiment intends to show an example where the classification task does not convey a deterministic scenario, that the convex IB Lagrangian can recover the IB curve in complex stochastic tasks with complex neural network architectures and that the value convergence can be employed to obtain a specific compression value even in stochastic settings where the IB curve is unknown.

Figure A2.

Results for the power IB Lagrangian in the MNIST dataset with $\alpha =\{0.5,1,2\}$, from top to bottom. In each row, from left to right it is shown (i) the information plane, where the region of possible solutions of the IB problem is shadowed in light orange and the information-theoretic limits are the dashed orange line; (ii) $I(T;Y)$ as a function of ${\beta }_u$; and (iii) the compression $I(X;T)$ as a function of ${\beta }_u$. In all plots, the red crosses joined by a dotted line represent the values computed with the training set, the blue dots the values computed with the validation set and the green stars the theoretical values computed as dictated by Proposition 3. Moreover, in all plots, it is indicated $I(X;Y)=H\left(Y\right)={log}_2\left(10\right)$ in a dashed, orange line. All values are shown in bits.

Figure A3.

Results for the exponential IB Lagrangian in the MNIST dataset with $\eta =\{log(2),1,1.5\}$, from top to bottom. In each row, from left to right it is shown (i) the information plane, where the region of possible solutions of the IB problem is shadowed in light orange and the information-theoretic limits are the dashed orange line; (ii) $I(T;Y)$ as a function of ${\beta }_u$; and (iii) the compression $I(X;T)$ as a function of ${\beta }_u$. In all plots, the red crosses joined by a dotted line represent the values computed with the training set, the blue dots the values computed with the validation set and the gren stars the theoretical values computed as dictated by Proposition 3. Moreover, in all plots, it is indicated $I(X;Y)=H\left(Y\right)={log}_2\left(10\right)$ in a dashed, orange line. All values are shown in bits.

Figure A4.

Depiction of the clusterization behavior of the bottleneck variable in the MNIST dataset. In the first row, from left to right, the power IB Lagrangian with different values of $\alpha =\{0.5,1,2\}$. In the second row, from left to right, the exponential IB Lagrangian with different values of $\eta =\{log(2),1,1.5\}$.

Figure A5.

Results for the exponential IB Lagrangian in the Fashion MNIST dataset with $\eta =1$. From left to right it is shown (i) the information plane, where the region of possible solutions of the IB problem is shadowed in light orange and the information-theoretic limits are the dashed orange line; (ii) $I(T;Y)$ as a function of ${\beta }_u$; and (iii) the compression $I(X;T)$ as a function of ${\beta }_u$. In all plots, the red crosses joined by a dotted line represent the values computed with the training set and the blue dots the values computed with the validation set. Moreover, in all plots, it is indicated $I(X;Y)=H\left(Y\right)={log}_2\left(10\right)$. All values are shown in bits.

Figure A6.

The top row shows the results for the normal IB Lagrangian, and the bottom row for the exponential IB Lagrangian with $\eta =1$, both in the California housing dataset. In each row, from left to right it is shown (i) the information plane, where the region of possible solutions of the IB problem is shadowed in light orange and the information-theoretic limits are the dashed orange line; (ii) $I(T;Y)$ as a function of ${\beta }_u$; and (iii) the compression $I(X;T)$ as a function of ${\beta }_u$. In all plots, the red crosses joined by a dotted line represent the values computed with the training set and the blue dots the values computed with the validation set. Moreover, in all plots, it is indicated $I(X;Y)$ as the empirical value obtained maximizing $I(T;Y)$ without compression limitations as in [26]. All values are shown in bits.

Figure A7.

The top row shows the results for the normal IB Lagrangian, and the bottom row for the power IB Lagrangian with $\alpha =0.1$, both in the TREC-6 dataset. In each row, from left to right it is shown (i) the information plane, where the region of possible solutions of the IB problem is shadowed in light orange and the information-theoretic limits are the dashed orange line; (ii) $I(T;Y)$ as a function of ${\beta }_u$; and (iii) the compression $I(X;T)$ as a function of ${\beta }_u$. In all plots, the red crosses joined by a dotted line represent the values computed with the training set and the blue dots the values computed with the validation set. Moreover, in all plots, it is indicated $H\left(Y\right)={log}_2\left(6\right)$. All values are shown in bits.

Appendix I. Guidelines for Selecting A Proper Function in the Convex IB Lagrangian

When choosing the right u function, it is important to find the right balance between avoiding value convergence and aiming for strong convexity. Practically, this balance is found by looking at how much faster u grows w.r.t. the identity function.

When the aim is not to draw the IB curve but to find a specific level of performance, we can exploit the value convergence phenomenon in order to design a stable performance targeted u function.

Appendix I.1. Avoiding Value Convergence

In order to explain this issue we are going to use the example of classification on MNIST [28], where $I(X;Y)=H\left(Y\right)={log}_2\left(10\right)$, and again the power and exponential IB Lagrangians.

If we use Proposition 3 on both Lagrangians we obtain the bijective mapping between their Lagrange multipliers and a certain level of compression in the classification setting:

1. Power IB Lagrangian: ${\beta }_{\mathrm{pow}}={\left((1+\alpha )I{(X;T)}^{\alpha }\right)}^{-1}$ and $I(X;T)={\left((1+\alpha ){\beta }_{\mathrm{pow}}\right)}^{-\frac{1}{\alpha }}$.

2. Exponential IB Lagrangian: ${\beta }_{\exp }={\left(\eta exp\left(\eta I\right(X;T\left)\right)\right)}^{-1}$ and $I(X;T)=-log\left(\eta {\beta }_{\exp }\right)/\eta$.

Hence, we can simply plot the curves of $I(X;T)$ vs. ${\beta }_u$ for different hyperparameters $\alpha$ and $\eta$ (see Figure A8). In this way, we can observe how increasing the growth of the function (e.g., increasing $\alpha$ or $\eta$ in this case) too much provokes that many different values of ${\beta }_u$ converge to very similar values of $I(X;T)$. This is an issue both for drawing the curve (for obvious reasons) and for aiming for a specific performance level. Due to the nature of the estimation of the IB Lagrangian, the theoretical and practical value of ${\beta }_u$ that yields a specific $I(X;T)$ may vary slightly (see Figure 1). Then if we select a function with too high growth, a small change in ${\beta }_u$ can result in a big change in the performance obtained.

Figure A8.

Theoretical bijection between $I(X;T)$ and different $\alpha$ from ${\beta }_{u,\min }$ to 1.5 in the power IB Lagrangian (top), and different $\eta$ in the domain $B_u$ in the exponential IB Lagrangian (bottom).

Appendix I.2. Aiming for Strong Convexity

Definition A2 ($\mathit{\mu }$-Strong Convexity).

If a function $f\left(r\right)$ is twice continuous differentiable and its domain is confined in the real line, then it is μ-strong convex if $f^{″}\left(r\right)\ge \mu \ge 0$$\forall r$.

Experimentally, we observed when the growth of our function $u\left(r\right)$ is small in the domain of interest $r>0$ the convex IB Lagrangian does not perform well (see first row of Figure A2 and Figure A3). Later we realized that this was closely related to the strength of the convexity of our function.

In Theorem 2 we imposed the function u to be strictly convex to enforce having a unique ${\beta }_u$ for each value of $I(X;T)$. Hence, since in practice we are not exactly computing the Lagrangian but an estimation of it (e.g., with the nonlinear IB [26]) we require strong convexity in order to be able to explore the IB curve.

We now look at the second derivative of the power and exponential function: $u^{″}\left(r\right)=(1+\alpha )\alpha r^{\alpha -1}$ and $u^{″}\left(r\right)={\eta }^{2}exp\left(\eta r\right)$ respectivelly. Here we see how both functions are inherently 0-strong convex for $r>0$ and $\alpha ,\eta >0$. However, values of $\alpha <1$ and $\eta <1$ could lead to low $\mu$-strong convexity in certain domains of r. Particularly, the case of $\alpha <1$ is dangerous because the function approaches 0-strong convexity as r increases, so the power IB Lagrangian performs poorly when low $\alpha$ are used to find high performances.

Appendix I.3. Exploiting Value Convergence

When the aim is not to draw or explore the IB curve, but to obtain a specific level of performance, the power of exponential IB Lagrangians aforementioned might not be the best choice due to the problems with value convergence or non-strong convexity. However, we can exploit the former in order to design a performance targeted u function.

For instance, if we look at Figure A8 we can see how a modification of the exponential IB Lagrangian could result in such a function. More precisely, a shifted exponential $u\left(r\right)=exp\left(\eta (r-r^{\ast })\right)$, with $\eta >0$ sufficiently large, converges to the compression level $r^{\ast }$. We can see this more clearly if we consider the shifted exponential IB Lagrangian ${\mathcal{L}}_{\mathrm{IB},\mathrm{sh}-\exp }(T;{\beta }_{\mathrm{sh}-\exp },\eta ,r^{\ast })=I(T;Y)-{\beta }_{\mathrm{sh}-\exp }exp\left(\eta (I(X;T)-r^{\ast })\right)$, since then the application of Proposition 3 results on $I(X;T)=-log(\eta {\beta }_{\mathrm{sh}-\exp }/f_{\mathrm{IB}}^{\prime }\left(I(X;T)\right))/\eta +r^{\ast }$, where $f_{\mathrm{IB}}^{\prime }\left(I(X;T)\right)$ is the derivative of $f_{\mathrm{IB}}$ evaluated at $I(X;T)$. We know $f_{\mathrm{IB}}^{\prime }=1$ in deterministic scenarios (Theorem 2) and that $f_{\mathrm{IB}}^{\prime }<1$ otherwise (see, e.g., [27]). Then, for large enough $\eta$, $I(X;T)\approx r^{\ast }$ regardless of the value of $f_{\mathrm{IB}}^{\prime }$.

For instance, if we consider a deterministic scenario like the MNIST dataset [28] with $I(X;Y)=H\left(Y\right)={log}_2\left(10\right)$, for $\eta =200$ and $r^{\ast }=2$ the range of the Lagrange multipliers that allow the exploration of the IB curve, according to Corollary 2, is ${\beta }_{\mathrm{sh}-\exp }\in [7.54\times {10}^{-178},2.61\times {10}^{171}]$. Furthermore, $I(X;T)$ is close to 2 for many values of ${\beta }_{\mathrm{sh}-\exp }$. For instance, $I(X;T)=1.974$ for ${\beta }_{\mathrm{sh}-\exp }=1$ and $I(X;T)=1.963$ for ${\beta }_{\mathrm{sh}-\exp }=8$. This ensures a stability in the performance level obtained so that small changes in the choice of ${\beta }_{\mathrm{sh}-\exp }$ do not result in significant changes on the performance (e.g., see top row from Figure 4).

If we now consider a stochastic scenario like the TREC-6 dataset [29] with $H\left(Y\right)={log}_2\left(6\right)$, for $\eta =200$ and $r^{\ast }=16$ the range of the Lagrange multipliers that allow the IB curve, according to Corollary 3, is ${\beta }_{\mathrm{sh}-\exp }\in [0,2.76{\left({inf}_{{\Omega }_x\subset \mathcal{X}}\left\{{\beta }_0\left({\Omega }_x\right)\right\}\right)}^{-1}\times {10}^{1287}]$, where ${\beta }_0$ and ${\Omega }_x$ are defined as in [27]. Then, unless ${\left({inf}_{{\Omega }_x\subset \mathcal{X}}\left\{{\beta }_0\left({\Omega }_x\right)\right\}\right)}^{-1}$ is of the order of ${10}^{-1287}$, the range of possible betas is wide. Moreover, $I(X;T)$ is close to 16 for many values of ${\beta }_{\mathrm{sh}-\exp }$. For example, $I(X;T)=15.939$ if $f_{\mathrm{IB}}^{\prime }=0.001$ at that point and $I(X;T)=15.973$ if $f_{\mathrm{IB}}^{\prime }=0.9$ for ${\beta }_{\mathrm{sh}-\exp }=1$; and $I(X;T)=15.929$ if $f_{\mathrm{IB}}^{\prime }=0.001$ at that point and $I(X;T)=15.963$ if $f_{\mathrm{IB}}^{\prime }=0.9$ for ${\beta }_{\mathrm{sh}-\exp }=8$. Hence, as in the deterministic scenario, the performance level obtained is stable with changes in the choice of ${\beta }_{\mathrm{sh}-\exp }$ (e.g., see bottom row from Figure 4).

Author Contributions

Conceptualization, B.R.G. and R.T.; formal analysis, B.R.G.; funding acquisition, M.S.; methodology, B.R.G. and R.T.; resources, M.S.; software, B.R.G.; supervision, R.T. and M.S.; visualization, B.R.G.; writing—original draft, B.R.G.; writing—review and editing, B.R.G., R.T. and M.S. All authors have read and agreed to the published version of the manuscript.

Funding

This work was supported in part by the Swedish Research Council.

Conflicts of Interest

The authors declare no conflict of interest.