A Novel Approach to the Partial Information Decomposition

Abstract

We consider the “partial information decomposition” (PID) problem, which aims to decompose the information that a set of source random variables provide about a target random variable into separate redundant, synergistic, union, and unique components. In the first part of this paper, we propose a general framework for constructing a multivariate PID. Our framework is defined in terms of a formal analogy with intersection and union from set theory, along with an ordering relation which specifies when one information source is more informative than another. Our definitions are algebraically and axiomatically motivated, and can be generalized to domains beyond Shannon information theory (such as algorithmic information theory and quantum information theory). In the second part of this paper, we use our general framework to define a PID in terms of the well-known Blackwell order, which has a fundamental operational interpretation. We demonstrate our approach on numerous examples and show that it overcomes many drawbacks associated with previous proposals.

1. Introduction

Understanding how information is distributed in multivariate systems is an important problem in many scientific fields. In the context of neuroscience, for example, one may wish to understand how information about an external stimulus is encoded in the activity of different brain regions. In computer science, one might wish to understand how the output of a logic gate reflects the information present in different inputs to that gate. Numerous other examples abound in biology, physics, machine learning, cryptography, and other fields [1,2,3,4,5,6,7,8,9,10].

Formally, suppose that we are provided with a random variable Y which we call the “target”, as well as a set of n random variables $X_1,\dots ,X_n$ which we call the “sources”. The partial information decomposition (PID), first proposed by Williams and Beer in 2010 [11], aims to quantify how information about the target is distributed among the different sources. In particular, the PID seeks to decompose the mutual information provided jointly by all sources into a set of nonnegative terms, such as redundancy (information present in each individual source), synergy (information only provided by the sources jointly, not individually), union information (information provided by at least one individual source), and unique information (information provided by only one individual source).

As discussed in detail below, the PID is inspired by an analogy between information theory and set theory. In this analogy, the information that the sources provide about the target are imagined as sets, while PID terms such as redundancy, union information, and synergy are imagined as the sizes of intersections, unions, and complements. While the analogy between information-theoretic and set-theoretic quantities is suggestive, it does not specify how to actually define the PID. Moreover, it has also been shown that existing measures from information theory (such as mutual information and conditional mutual information) cannot be used directly to construct the PID, since these measures conflate contributions from different terms like synergy and redundancy [11,12]. In response, many proposals for how to define PID terms have been advanced [5,13,14,15,16,17,18,19,20,21]. However, existing proposals suffer from various drawbacks, such as behaving counterintuitively on simple examples, being limited to only two sources, or lacking a clear operational interpretation. Today there is no generally agreed-upon way of defining the PID.

In this paper, we propose a new and principled approach to the PID which addresses these drawbacks. Our approach can handle any number of sources and can be justified in algebraic, axiomatic, and operational terms. We present our approach in two parts.

In part I (Section 4), we propose a general framework for defining the PID. Our framework does not prescribe specific definitions, but instead shows how an information-theoretic decomposition can be grounded in a formal analogy with set theory. Specifically, we consider the definitions of “set intersection” and “set union” in set theory: the intersection of sets $S_1,S_2,\dots$ is the largest set that is contained in all of the $S_i$, while the union of sets $S_1,S_2,\dots$ is the smallest set that contains all of the $S_i$. As we show, these set-theoretic definitions can be mapped into information-theoretic terms by treating “sets” as random variables, “set size” as mutual information between a random variable and the target Y, and “set inclusion” as some externally specified ordering relation ⊏, which specifies when one random variable is more informative than another. Using this mapping, we define information-theoretic redundancy and union information in the same way that the sizes of intersections and unions are defined in set theory (other PID terms, such as synergy and unique information, can be computed in a straightforward way from redundancy and union information). Moreover, while our approach is motivated by set-theoretic intuitions, as we show in Section 4.2, it can also be derived from an alternative axiomatic foundation. We finish part I by reviewing relevant prior work in information theory and the PID literature. We also discuss how our framework can be generalized beyond the standard setting of the PID and even beyond Shannon information theory, to domains like algorithmic information theory and quantum information theory.

One unusual aspect of our framework is that it provides independent definitions of union information and redundancy. Most prior work on the PID has focused exclusively on the definition of redundancy, because it assumed that union information can be determined from redundancy using the so-called “inclusion-exclusion principle”. In Section 4.3, we argue that the inclusion-exclusion principle should not be expected to hold in the context of the PID.

Part I provides a general framework. Concrete definitions of the PID can be derived from this general framework by choosing a specific “more informative” ordering relation ⊏. In fact, the study of ordering relations between information sources has a long history in statistics and information theory [22,23,24,25,26,27]. One particularly important relation is the so-called “Blackwell order” [13,28], which has a fundamental operational interpretation in terms of utility maximization in decision theory.

In part II of this paper (Section 5), we combine the general framework developed in part I with the Blackwell order. This gives rise to concrete definitions of redundancy and union information. We show that our measures behave intuitively and have simple operational interpretations in terms of decision theory. Interestingly, while our measure of redundancy is novel, our measure of union information has previously appeared in the literature under a different guise [13,17].

In Section 6, we compare our redundancy measure to previous proposals, and illustrate it with various bivariate and multivariate examples. We finish the paper with a discussion and proposals for future work in Section 7.

We introduce some necessary notation and preliminaries in the next section. In addition, we provide background regarding the PID in Section 3. All proofs, as well as some additional results, are found in the appendix.

2. Notation and Preliminaries

We use uppercase letters ($Y,X,Q,\dots$) to indicate random variables over some underlying probability space. We use lowercase letters ($y,x,q,\dots$) to indicate specific outcomes of random variables, and calligraphic letters ($\mathcal{Y},\mathcal{X},\mathcal{Q}\dots$) to indicate sets of outcomes. We often index random variables with a subscript, e.g., the random variable $X_i$ with outcomes $x_i\in {\mathcal{X}}_i$ (so $x_i$ does not refer to the $i^{\mathrm{th}}$ outcome of random variable X, but rather to some generic outcome of random variable $X_i$). We use notation like $A-B-C$ to indicate that A is conditionally independent of C given B. Except where otherwise noted, we assume that all random variables have a finite number of outcomes.

We use notation like $P_X\left(x\right)$ to indicate the probability distribution associated with random variable X, $P_{XY}(x,y)$ to indicate the joint probability distribution associated with random variables X and Y, and $P_{X|Y}\left(x\right|y)$ to indicate the conditional probability distribution of X given Y. Given two random variables X and Y with outcome sets $\mathcal{X}$ and $\mathcal{Y}$, we use notation like ${\kappa }_{X|Y}\left(x\right|y)$ to indicate some stochastic channel of outputs $x\in \mathcal{X}$ given inputs $y\in \mathcal{Y}$. In general, a channel ${\kappa }_{X|Y}$ specifies some arbitrary conditional distribution of X given Y, which can be different from $P_{X|Y}$, the actual conditional distribution of X given Y (as determined by the underlying probability space).

As described above, we consider the information that a set of “source” random variables $X_1,\dots ,X_n$ provide a “target” random variable Y. Without loss of generality, we assume that the marginal distributions $P_Y$ and $P_{X_i}$ for all i have full support (if they do not, one can restrict $\mathcal{Y}$ and/or ${\mathcal{X}}_i$ to outcomes that have strictly positive probability).

Finally, note that despite our use of the terms “source” and “target”, we do not assume any causal directionality between the sources and target (see also discussion in [29]). For example, in neuroscience, Y might be an external stimulus which causes the activity of brain regions $X_1,\dots ,X_n$, while in computer science Y might represent the output of a logic gate caused by inputs $X_1,\dots ,X_n$ (so the causal direction is reversed). In yet other contexts, there could be other causal relationships among $X_1,\dots ,X_n$ and Y, or they might not be causally related at all.

3. Background on the Partial Information Decomposition (PID)

Given a set of sources $X_1,\dots ,X_n$ and a target Y, the PID aims to decompose $I(Y;X_1,\dots ,X_n)$, the total mutual information provided by all sources about the target, into a set of nonnegative terms such as [11,12]:

• Redundancy $I_{\cap }(X_1;\dots ;X_n\to Y)$, the information present in each individual source. Redundancy can be considered as the intersection of the information provided by different sources and is sometimes called “intersection information” in the literature [16,18].

• Union information $I_{\cup }(X_1;\dots ;X_n\to Y)$, the information provided by at least one individual source [12,17].

• Synergy $S(X_1;\dots ;X_n\to Y)$, the information found in the joint outcome of all sources, but not in any of their individual outcomes. Synergy is defined as [17]

  $$\begin{array}{cc}\hfill S(X_1;\dots ;X_n\to Y)& =I(Y;X_1,\dots ,X_n)-I_{\cup }(X_1;\dots ;X_n\to Y).\hfill \end{array}$$ (1)
• Unique information in source $X_i$, $U(X_i\to Y|X_1;\dots ;X_n)$, the non-redundant information in each particular source. Unique information is defined as

  $$\begin{array}{cc}\hfill U(X_i\to Y|X_1;\dots ;X_n)& =I(Y;X_i)-I_{\cap }(X_1;\dots ;X_n\to Y).\hfill \end{array}$$ (2)

In addition to the above terms, one can also define excluded information,

$$E(X_i\to Y|X_1;\dots ;X_n)=I_{\cup }(X_1;\dots ;X_n\to Y)-I(Y;X_i),$$ (3)

as the information in the union of the sources which is not in a particular source $X_i$. To our knowledge, excluded information has not been previously considered in the PID literature, although it is the natural “dual” of unique information as defined in Equation (2).

Given the definitions above, once a measure of redundancy $I_{\cap }$ is chosen, unique information is determined by Equation (2). Similarly, once a measure of union information $I_{\cup }$ is chosen, synergy and excluded information are determined by Equations (1) and (3). In Figure 1, we illustrate the relationships between these different PID terms for the simple case of two sources, $X_1$ and $X_2$. We show two different decompositions of the information provided by the sources jointly, $I(X_1,X_2;Y)$, and individually, $I(X_1;Y)$ and $I(X_2;Y)$. The diagram on the left shows the decomposition defined in terms of redundancy $I_{\cap }$, while the diagram on the right shows the decomposition defined in terms of union information $I_{\cup }$.

Figure 1.

Partial information decomposition of the information provided by two sources about a target. On the left, we show the decomposition induced by redundancy $I_{\cap }$, which leads to measures of unique information U. On the right, we show the decomposition induced by union information $I_{\cup }$, which leads to measures of synergy S and excluded information E.

When more than two sources are present, the PID can be used to define additional terms, beyond the ones shown in Figure 1. For example, for three sources, one can define redundancy terms like $I_{\cap }(X_1,X_2,X_3\to Y)$ (representing the information found in all individual sources) as well as redundancy terms like $I_{\cap }((X_1,X_2),(X_1,X_3),(X_2,X_3)\to Y)$ (representing the information found in all pairs of sources), and similarly for union information.

The idea that redundancy and union information lead to two different information decompositions is rarely discussed in the literature. In fact, the very concept of union information is rarely discussed in the literature explicitly (although it often appears in an implicit form via measures of synergy, since synergy is related to union information through Equation (1)). As we discuss below in Section 4.3, the reason for this omission is that most existing work assumes (whether implicitly or explicitly) that redundancy and union information are not independent measures, but are instead related via the so-called “inclusion-exclusion principle”. If the inclusion-exclusion principle is assumed to hold, then the distinction between the two decompositions disappears. We discuss this issue in greater detail below, where we also argue that the inclusion-exclusion principle should not be expected to hold in the context of the PID.

We have not yet described how the redundancy and union information measures $I_{\cap }$ and $I_{\cup }$ are defined. In fact, this remains an open research question in the field (and one which this paper will address). When they first introduced the idea of the PID, Williams and Beer proposed a set of intuitive axioms that any measure of redundancy should satisfy [11,12], which we summarize in Appendix A. In later work, Griffith and Koch [17] proposed a similar set of axioms that union information should satisfy, which are also summarized in Appendix A. However, these axioms do not uniquely identify a particular measure of redundancy or union information.

Williams and Beer also proposed a particular redundancy measure which satisfies their axioms, which we refer to as $I_{\cap }^{\mathrm{WB}}$ [11,12]. Unfortunately, $I_{\cap }^{\mathrm{WB}}$ has been shown to behave counterintuitively in some simple cases [19,20]. For example, consider the so-called “COPY gate”, where there are two sources $X_1$ and $X_2$ and the target is a copy of their joint outcomes, $Y=(X_1,X_2)$. If $X_1$ and $X_2$ are statistically independent, $I(X_1;X_2)=0$, then intuition suggests that the two sources provide independent information about Y and therefore that redundancy should be 0. In general, however, $I_{\cap }^{\mathrm{WB}}(X_1;X_2\to Y)$ does not vanish in this case. To avoid this issue, Ince [20] proposed that any valid redundancy measure should obey the following property:

$$\mathrm{If}I(X_1;X_2)=0,\mathrm{then}{I}_{\cap }(X_1;X_2\to (X_1,X_2))=0,$$ (4)

which is called the Independent identity property.

In recent years, many other redundancy measures have been proposed [13,15,16,18,19,20,21]. However, while some of these proposals satisfy the Independent identity property, they suffer various other drawbacks, such as exhibiting other types of counterintuitive behavior, being limited to two sources, and/or lacking a clear operational motivation. We discuss some of these previously proposed measures in Section 4.4, Section 5.4 and Section 6.

Unlike redundancy, to our knowledge only two measures of union information have been advanced. The first one appeared in the original work on the PID [12], and was derived from $I_{\cap }^{\mathrm{WB}}$ using the inclusion-exclusion principle. The second one appeared more recently [13,17] and is discussed in Section 5.4 below.

4. Part I: Redundancy and Union Information from an Ordering Relation

4.1. Introduction

As mentioned above, PID is motivated by an informal analogy with set theory [12]. In particular, redundancy is interpreted analogously to the size of the intersection of the sources $X_1,\dots ,X_n$, while union information is interpreted analogously to the size of their union.

We propose to define the PID by making this analogy formal, and in particular by going back to the algebraic definitions of intersection and union in set theory. In pursuing this direction, we build on a line of previous work in information theory and PID, which we discuss in Section 4.4.

Recall that in set theory, the intersection of sets $S_1,\dots ,S_n\subseteq U$ (where U is some universal set) is the largest set that is contained in all $S_i$ (Section 7.2, [30]). This means that the size of the intersection can be written as

$$\begin{array}{cc}\hfill \left|\bigcap _i{S}_i\right|& =\underset{T\subseteq U}{\sup }\left|T\right|\hspace{1em}\mathrm{such}\mathrm{that}\forall iT\subseteq S_i,\hfill \end{array}$$ (5)

Similarly, the union of sets $S_1,\dots ,S_n\subseteq U$ is the smallest set that contains all $S_i$ (Section 7.2, [30]), so the size of the union can be written as

$$\begin{array}{cc}\hfill \left|\bigcup _i{S}_i\right|& =\underset{T\subseteq U}{\inf }\left|T\right|\hspace{1em}\mathrm{such}\mathrm{that}\forall i{S}_i\subseteq T.\hfill \end{array}$$ (6)

Equations (5) and (6) are useful because they express the size of the intersection and union via an optimization over simpler terms (the size of individual sets, $\left|T\right|$, and the subset inclusion relation, ⊆).

We translate these definitions to the information-theoretic setting of the PID. We take the analogue of a “set” to be some random variable A that provides information about the target Y, and the analogue of “set size” to be the mutual information $I(A;Y)$. In addition, we assume that there is some ordering relation ⊏ between random variables analogous to set inclusion ⊆. Given such a relation, the expression $A⊏B$ means that random variable B is “more informative” than A, in the sense that the information that A provides about Y is contained within the information that B provides about Y.

At this point, we leave the ordering relation ⊏ unspecified. In general, we believe that the choice of ⊏ will not be determined from purely information-theoretic considerations, but may instead depend on the operational setting and scientific domain in which the PID is applied. At the same time, there has been a great deal of research on ordering relations in statistics and information theory. In part II of this paper, Section 5, we will combine our general framework with a particular ordering relation, the so-called “Blackwell order”, which has a fundamental interpretation in terms of decision theory.

We now provide formal definitions of redundancy and union information, relative to the choice of ordering relation ⊏. In analogy to Equation (5), we define redundancy as

$$\begin{array}{cc}\hfill I_{\cap }(X_1;\dots ;X_n\to Y)& :=\underset{Q}{\sup }I(Q;Y)\hspace{1em}\mathrm{such}\mathrm{that}\forall iQ⊏X_i\hfill \end{array}$$ (7)

where the maximization is over all random variables with a finite number of outcomes. Thus, redundancy $I_{\cap }$ is the maximum information about Y in any random variable that is less informative than all of the sources. In analogy with Equation (6), we define union information as

$$\begin{array}{cc}\hfill I_{\cup }(X_1;\dots ;X_n& \to Y):=\underset{Q}{\inf }I(Q;Y)\hspace{1em}\mathrm{such}\mathrm{that}\forall i{X}_i⊏Q\hfill \end{array}$$ (8)

Thus, union information $I_{\cup }$ is the minimum information about Y in any random variable that is more informative than all of the sources. Given these definitions, other elements of the PID (such as unique information, synergy, and excluded information) can be defined using the expressions found in Section 3. Note that $I_{\cap }$ and $I_{\cup }$ depend the choice of ordering relation ⊏, although for convenience we leave this dependence implicit in our notation.

One of the attractive aspects of our definitions is that they do not simply quantify the amount of redundancy and union information, but also specify the “content” of that redundant and union information. In particular, the random variable Q that achieves the optimum in Equation (7) specifies the content of the redundant information via the joint distribution $P_{YQ}$. Similarly, the random variable Q which achieves the optimum in Equation (8) specifies the content of the union information via the joint distribution $P_{YQ}$. Note that these optimizing Q may not be unique, reflecting the fact that there may be different ways to represent the redundancy or union information. (Note also that the supremum or infinitum may not be achieved in Equations (7) and (8), in which case one can consider Q that achieve the optimal values to any desired precision $ϵ>0$.)

So far we have not made any assumptions about the ordering relation ⊏. However, we can derive some useful bounds by introducing three weak assumptions:

1. Monotonicity of mutual information: $A⊏B\Rightarrow I(A;Y)\le I(B;Y)$ (less informative sources have less mutual information).

2. Reflexivity: $A⊏A$ for all A (each source is at least as informative as itself).

3. For all sources $X_i$, $O⊏X_i⊏(X_1,\dots ,X_n)$, where O indicates a constant random variable with a single outcome and $(X_1,\dots ,X_n)$ indicates all sources considered jointly (each source is more informative than a trivial source and less informative than all sources jointly).

Assumptions I and II imply that the redundancy and union information of a single source are equal to the mutual information in that source:

$$I_{\cap }(X_1\to Y)=I_{\cup }(X_1\to Y)=I(X_1;Y).$$

Assumptions I and III imply the following bounds on redundancy and union information:

$$\begin{array}{cc}\hfill 0& \le I_{\cap }(X_1;\dots ;X_n\to Y)\le \underset{i}{\min }I(Y;X_i).\hfill \end{array}$$ (9)

$$\begin{array}{cc}\hfill \underset{i}{\max }I(Y;X_i)& \le I_{\cup }(X_1;\dots ;X_n\to Y)\le I(Y;X_1,\dots ,X_n).\hfill \end{array}$$ (10)

Equation (9) in turn implies that the unique information in each source $X_i$, as defined in Equation (2), is bounded between 0 and $I(Y;X_i)$. Similarly, Equation (10) implies that the synergy, as defined in Equation (1), obeys

$$0\le S(X_1;\dots ;X_n\to Y)\le \underset{i}{\min }I(Y;X_1,\dots ,X_n|X_i),$$

where we have used the chain rule $I(Y;X_1,\dots ,X_n)=I(Y;X_i)+I(Y;X_1,\dots ,X_n|X_i)$. Equation (10) also implies that excluded information in each source $X_i$, as defined in Equation (3), is bounded between 0 and $I(Y;X_1,\dots ,X_n|X_i)$.

Note that in general, stronger orders give smaller values of redundancy and larger values of union information. Consider two orders ⊏ and ${⊏}^{\prime }$ where the first one is stronger than the second: $A⊏B\Rightarrow A{⊏}^{\prime }B$ for all A and B. Then, any Q in the feasible set of Equation (7) under ⊏ will also be in the feasible set under ${⊏}^{\prime }$, and similarly for Equation (8). Therefore, $I_{\cap }$ defined relative to ⊏ will have a lower value than $I_{\cap }$ defined relative to ${⊏}^{\prime }$, and vice versa for $I_{\cup }$.

In the rest of this section, we discuss alternative axiomatic justifications for our general framework, the role of the inclusion-exclusion principle, relation to prior work, and further generalizations. Readers who are more interested in the use of our framework to define concrete measures of redundancy and union information may skip to Section 5.

4.2. Axiomatic Derivation

In Section 4.1, we defined the PID in terms of an algebraic analogy with intersection and union in set theory. This definition can be considered as the primary one in our framework. At the same time, the same definitions can also be derived in an alternative manner from a set of axioms, as commonly sought after in the PID literature. In particular, in Appendix B, we prove the following result regarding redundancy.

Theorem 1. 

Any redundancy measure that satisfies the following five axioms is equal to $I_{\cap }(X_1;\dots ;X_n\to Y)$ as defined in Equation (7).

1. Symmetry: $I_{\cap }(X_1;\dots ;X_n\to Y)$ is invariant to the permutation of $X_1,\dots ,X_n$.

2. Self-redundancy: $I_{\cap }(X_1\to Y)=I(Y;X_1)$.

3. Monotonicity: $I_{\cap }(X_1;\dots ;X_n\to Y)\le I_{\cap }(X_1;\dots ;X_{n-1}\to Y)$.

4. Order equality: $I_{\cap }(X_1;\dots ;X_n\to Y)=I_{\cap }(X_1;\dots ;X_{n-1}\to Y)$ if $X_i⊏X_n$ for some $i<n$.

5. Existence: There is some Q such that $I_{\cap }(X_1;\dots ;X_n\to Y)=I(Y;Q)$ and $Q⊏X_i$ for all i.

While Symmetry, Self-redundancy, and Monotonicity axioms are standard in the PID literature (see Appendix A), the last two axioms require some explanation. Order equality is a generalization of the previously proposed Deterministic equality axiom, described in Appendix A, where the condition $X_i=f\left(X_n\right)$ (deterministic relationship) is generalized to the “more informative” relation $X_i⊏X_n$. This axiom reflects the idea that if a new source $X_n$ is more informative than an existing source $X_i$, then redundancy shouldn’t decrease when $X_n$ is added.

Existence is the most novel of our proposed axioms. It says that for any set of sources $X_1,\dots ,X_n$, there exists some random variable which captures the redundant information. It is similar to the statement in axiomatic set theory that the intersection of a collection of sets is itself a set (note that in Zermelo-Fraenkel set theory, this statement is derived from the Axiom of Separation).

We can derive a similar result for union information (proof in Appendix B).

Theorem 2. 

Any union information measure that satisfies the following five axioms is equal to $I_{\cup }(X_1;\dots ;X_n\to Y)$ as defined in Equation (8).

1. Symmetry: $I_{\cup }(X_1;\dots ;X_n\to Y)$ is invariant to the permutation of $X_1,\dots ,X_n$.

2. Self-union: $I_{\cup }(X_1\to Y)=I(Y;X_1)$.

3. Monotonicity: $I_{\cup }(X_1;\dots ;X_n\to Y)\ge I_{\cup }(X_1;\dots ;X_{n-1}\to Y)$.

4. Order equality: $I_{\cup }(X_1;\dots ;X_n\to Y)=I_{\cup }(X_1;\dots ;X_{n-1}\to Y)$ if $X_n⊏X_i$ for some $i<n$.

5. Existence: There is some Q such that $I_{\cup }(X_1;\dots ;X_n\to Y)=I(Y;Q)$ and $X_i⊏Q$ for all i.

These axioms are dual to the redundancy axioms outlined above. Compared to previously proposed axioms for union information, as described in Appendix A, the most unusual of our axioms is Existence. It says that given a set of sources $X_1,\dots ,X_n$, there exists some random variable which captures the union information. It is similar in spirit to the “Axiom of Union” in axiomatic set theory [31].

Finally, note that for some choices of ⊏, there may not exist measures of redundancy and/or union information that satisfy the axioms in Theorems 1 and 2, in which case these theorems still hold but are trivial. However, even in such “pathological” cases, $I_{\cap }$ and $I_{\cup }$ can still be defined via Equations (7) and (8), as long as ⊏ has a “least informative” and a “most informative” element (e.g., as provided by Assumption III above), so that the feasible sets are not empty. In this sense, the definitions in Equations (7) and (8) are more general than the axiomatic derivations provided by Theorems 1 and 2.

4.3. Inclusion-Exclusion Principle

One unusual aspect of our approach is that, unlike most previous work, we propose separate measures of redundancy and union information.

Recall that in set theory, the size of the intersection and the union are not independent of each other, but are instead related by the inclusion-exclusion principle (IEP). For example, given any two sets S and T, the IEP states that the size of the union of S and T is given by the sum of their individual sizes minus the intersection,

$$\left|S\cup T\right|=\left|S\right|+\left|T\right|-\left|S\cap T\right|.$$ (11)

More generally, the IEP relates the sizes of intersection and unions for any number of sets, via the following inclusion-exclusion formulas:

$$\begin{array}{cc}\hfill \left|\bigcup _{i=1}^n{S}_i\right|& =\sum _{⌀\ne J\subseteq \{1,\dots ,n\}}{(-1)}^{\left|J\right|-1}\left|\bigcap _{i\in J}{S}_i\right|.\hfill \end{array}$$ (12)

$$\begin{array}{cc}\hfill \left|\bigcap _{i=1}^n{S}_i\right|& =\sum _{⌀\ne J\subseteq \{1,\dots ,n\}}{(-1)}^{\left|J\right|-1}\left|\bigcup _{i\in J}{S}_i\right|.\hfill \end{array}$$ (13)

Historically, the IEP has played an important role in analogies between set theory and information theory, which began to be explored in 1950s and 1960s [32,33,34,35,36]. Recall that the entropy $H\left(X\right)$ quantifies the amount of information gained by learning the outcome of random variable X. It has been observed that, for a set of random variables $X_1,\dots ,X_n$, the joint entropy $H(X_1,\dots ,X_n)$ behaves somewhat like the size of the union of the information in the individual variables. For instance, like the size of the union, joint entropy is subadditive ($H\left(X_1\right)+H\left(X_2\right)\ge H(X_1,X_2)$) and increases with additional random variables ($H(X_1,X_2)\ge H\left(X_1\right)$). Moreover, for two random variables $X_1$ and $X_2$, the mutual information $I(X_1;X_2)=H\left(X_1\right)+H\left(X_2\right)-H(X_1,X_2)$ acts like the size of the intersection of the information provided by $X_1$ and $X_2$, once intersection is defined analogously to the IEP expression in Equation (11) [35,36]. Given the general IEP formula in Equation (13), this can be used to define the size of the intersection between any number of random variables. For instance, the size of a three-way intersection is

$$\begin{array}{cc}\hfill I(X_1;X_2;X_3)=& H\left(X_1\right)+H\left(X_2\right)+H\left(X_3\right)\hfill \\ \hfill & \hspace{1em}-H(X_1,X_2)-H(X_1,X_3)-H(X_2,X_3)+H(X_1,X_2,X_3),\hfill \end{array}$$

a quantity called co-information or interaction information in the literature [32,33,35,36,37].

Unfortunately, interaction information, as well as other higher-order interaction terms defined via the IEP, can take negative values [32,35,37]. This conflicts with the intuition that information measures should always be non-negative, in the same way that set size is always non-negative.

One of the primary motivations for the PID, as originally proposed by Williams and Beer [11,12], was to solve the problem of negativity encountered by interaction information. To develop a non-negative information decomposition, Williams and Beer took two steps. First, they considered the information that a set of sources $X_1,\dots ,X_n$ provide about some target random variable Y. Second, they developed a non-negative measure of redundancy ($I_{\cap }^{\mathrm{WB}}$) which leads to a non-negative union information once an IEP formula like Equation (12) is applied (Theorem 4.7, [12]). For example, in the original proposal, union information and redundancy are related via

$$I_{\cup }(X_1;X_2\to Y)\stackrel{?}{=}I(Y;X_1)+I(Y;X_2)-I_{\cap }(X_1;X_2\to Y),$$ (14)

which is the analogue of Equation (11). This can be plugged into expressions like Equation (1), so as to express synergy in terms of redundancy as

$$S(X_1;\dots ;X_n\to Y)\stackrel{?}{=}I(Y;X_1,\dots ,X_n)-I(Y;X_1)-I(Y;X_2)+I_{\cap }(X_1;X_2\to Y).$$ (15)

The meaning of IEP-based identities such as Equations (14) and (15) can be illustrated using the Venn diagrams in Figure 1. In particular, they imply that the pink region in the right diagram is equal in size to the pink region in the left diagram, and that the grey region in the left diagram is equal in size to the grey region in the right diagram. More generally, IEP implies an equivalence between the information decomposition based on redundancy and the one based on union information.

As mentioned in Section 3, due to shortcomings in the original redundancy measure $I_{\cap }^{\mathrm{WB}}$, numerous other proposals for the PID have been advanced. Most of these proposals introduce new measures of redundancy, while keeping the general structure of the PID as introduced by Williams and Beer. In particular, most of these proposals assume that the IEP holds, so that union information can be derived from a measure of redundancy. While the assumption of the IEP is sometimes stated explicitly, more frequently it is implicit in the definitions used. For example, many proposals assume that synergy is related to redundancy via an expression like Equation (15), although (as shown above) this implicitly assumes that the IEP holds. In general, the IEP has been largely an unchallenged and unexamined assumption in the PID field. It is easy to see the appeal of the IEP: it builds on deep-seated intuitions about intersection/union from set theory and Venn diagrams, it has a long history in the information-theoretic literature, and it simplifies the problem of defining the PID since it only requires a measure of redundancy to be defined—rather than a measure of redundancy and a measure of union information. (Note that one can also start from union information and then derive redundancy via the IEP formula in Equation (13), as in Appendix B of Ref. [17], although this is much less common in the literature.)

However, there is a different way to define a non-negative PID, which is still grounded in a formal analogy with set theory but does not assume the IEP. Here, one defines measures of redundancy and union information based on the underlying algebra of intersection and union: the intersection of $X_1,\dots ,X_n$ is the largest element that is less than each $X_i$, while the union is the smallest element that is greater than each $X_i$. Given these definitions, intersections and unions are not necessarily related to each numerically, as in the IEP, but are instead related by an algebraic duality.

This latter approach is the one we pursue in our definitions (it has also appeared in some prior work, which we review in the next subsection). In general, the IEP will not hold for redundancy and union information as defined in Equations (7) and (8). (To emphasize this point, we put a question mark in Equations (14) and (15), and made the sizes of the pink and grey regions visibly different in Figure 1). However, given the algebraic and axiomatic justifications for $I_{\cap }$ and $I_{\cup }$, we do not see the violation of the IEP as a fatal issue. In fact, there are many domains where generalizations of intersections and unions do not obey the IEP. For example, it is well-known that the IEP is violated in the domain of vector spaces, once the size of a vector space is measured in terms of its dimension [38]. The PID is simply another domain where the IEP should not be expected to hold.

We believe that many problems encountered in previous work on the PID—such as the failure of certain redundancy measures to generalize to more than two sources, or the appearance of uninterpretable negative synergy values—are artifacts of the IEP assumption. In fact, the following result shows that any measures of redundancy and union information which satisfy several reasonable assumptions must violate the IEP as soon as 3 or more sources are present (the proof, in Appendix I, is based on a construction from [39,40]).

Lemma 1. 

Let $I_{\cap }$ be any nonnegative redundancy measure which obeys Symmetry, Self-redundancy, Monotonicity, and Independent identity. Let $I_{\cup }$ be any union information measure which obeys $I_{\cup }(X_1;\dots ;X_n\to Y)\le I(Y;X_1,\dots ,X_n)$. Then, $I_{\cap }$ and $I_{\cup }$ cannot be related by the inclusion-exclusion principle for 3 or more sources.

The idea that different information decompositions may arise from redundancy versus synergy (and therefore union information) has recently appeared in the PID literature [15,40,41,42,43]. In particular, Chicharro and Panzeri proposed a PID that involves two decomposition: an “information gain” decomposition based on redundancy and an “information loss” decomposition based on synergy [41]. These decompositions correspond to the two Venn diagrams shown in Figure 1.

4.4. Relation to Prior Work

Here we discuss prior work which is relevant to our algebraic approach to the PID.

First, note that our definitions of redundancy and union information in Equations (7) and (8) are closely related to notions of “meet” and “join” in a field of algebra called order theory, which generalize intersections and unions to domains beyond set theory [44]. Given a set of objects S and an order ⊏, the meet of $a,b\in S$ is the unique largest $c\in S$ that is smaller than both a and b: $c⊏a,c⊏b$ and $d⊏c$ for any d that obeys $d⊏a,d⊏b$. Similarly, the join of $a,b\in S$ is the unique smallest c that is larger than both a and b: $a⊏c,a⊏c$ and $c⊏d$ for any d that obeys $a⊏d,b⊏d$. Note that meets and joins are only defined when ⊏ is a special type of partial order called a lattice. This is a strict requirement, and many important ordering relations in information theory are not lattices (this includes the “Blackwell order”, which we will consider in part II of this paper [45]).

In our approach, we do not require the ordering relation ⊏ to be a lattice, or even a partial order. We do not require these properties because we do not aim to find the unique union random variable or the unique redundancy random variable. Instead, we aim to quantify the size of the intersection and the size of the union, which we do by optimizing mutual information subject to constraints, as Equations (7) and (8). These definitions are well-defined even when ⊏ is not a lattice, which allows us to consider a much broader set of ordering relations.

We mention three important precursors of our approach that have been proposed in the PID literature. First, Griffith et al. [16] considered the following order between random variables:

$$A⊲B\mathrm{iff}A=f\left(B\right)\mathrm{for}\mathrm{some}\mathrm{deterministic}\mathrm{function}f.$$ (16)

This ordering relation ⊲ was first considered in a 1953 paper by Shannon [22], who showed that it defines a lattice over random variables. That paper was the first to introduce the algebraic idea of meets and joins into information theory, leading to an important line of subsequent research [46,47,48,49,50]. Using this order, Ref. [16] defined redundancy as the maximum mutual information in any random variable that is a deterministic function of all of the sources,

$$I_{\cap }^{⊲}(X_1;\dots ;X_n\to Y):=\underset{Q}{\max }I(Q;Y)\hspace{1em}\mathrm{such}\mathrm{that}\hspace{1em}\forall iQ⊲X_i,$$ (17)

which is clearly a special case of Equation (7). Unfortunately, in practice, $I_{\cap }^{⊲}$ is not a useful redundancy measure, as it tends to give very small values and is highly discontinuous. For example, $I_{\cap }^{⊲}(X_1;\dots ;X_n\to Y)=0$ whenever the joint distribution $P_{X_1\dots X_{n}Y}$ has full support, meaning that it vanishes on almost all joint distributions [16,18,47]. The reason for this counterintuitive behavior is that the order ⊲ formalizes an extremely strict notion of “more informative”, which is not robust to noise.

Given the deficiencies of $I_{\cap }^{⊲}$, Griffith and Ho [18] proposed another measure of redundancy (also discussed as $I_{\cap }^2$ in Ref. [49]),

$$I_{\cap }^{\mathrm{GH}}(X_1;\dots ;X_n\to Y):=\underset{Q}{\max }I(Q;Y)\hspace{1em}\mathrm{such}\mathrm{that}\hspace{1em}\forall iQ-X_i-Y.$$ (18)

This measure is also a special case of Equation (7), where the more informative relation $A⊏B$ is formalized via the conditional independence condition $A-B-Y$. This measure is similar to the redundancy measure we propose in part II of this paper, and we discuss it in more detail in Section 5.4. (Note that there are some incorrect claims about $I_{\cap }^{\mathrm{GH}}$ in the literature: Lemmas 6 and 7 of Ref. [49] incorrectly state that $I_{\cap }^{\mathrm{GH}}(X_1;X_2\to Y)=0$ whenever $X_1$ and $X_2$ are independent—see the AND gate counterexample in Section 6—while Ref. [18] incorrectly states that $I_{\cap }^{\mathrm{GH}}$ obeys a property called Target Monotonicity).

Finally, we mention the so-called “minimum mutual information” redundancy $I_{\cap }^{\mathrm{MMI}}$ [51]. This is perhaps the simplest redundancy measure, being equal to the minimal mutual information in any source: $I_{\cap }^{\mathrm{MMI}}(X_1;\dots ;X_n\to Y):={\min }_{i}I(X_i;Y)$. It can be written in the form of Equation (7) as

$$I_{\cap }^{\mathrm{MMI}}(X_1;\dots ;X_n\to Y):=\underset{Q}{\max }I(Q;Y)\hspace{1em}\mathrm{such}\mathrm{that}\hspace{1em}\forall iI(Q;Y)\le I(X_i;Y).$$ (19)

This redundancy measure has been criticized for depending only on the amount of information provided by the different sources, being completely insensitive to the content of that information. Nonetheless, $I_{\cap }^{\mathrm{MMI}}$ can be useful in some settings, and it plays an important role in the context of Gaussian random variables [51].

Interestingly, unlike $I_{\cap }^{\mathrm{MMI}}$, the original redundancy measure proposed by Williams and Beer [11], $I_{\cap }^{\mathrm{WB}}$, does not appear to be a special case of Equation (7) (at least not under the natural definition of the ordering relation ⊏). We demonstrate this using a counter-example in Appendix H.

As mentioned in Section 4.1, stronger ordering relations give smaller values of redundancy. For the orders considered above, it is easy to show that

$$A⊲B\Rightarrow A-B-Y\Rightarrow I(A;Y)\le I(B;Y).$$ (20)

This implies that $I_{\cap }^{⊲}\le I_{\cap }^{\mathrm{GH}}\le I_{\cap }^{\mathrm{MMI}}$. In fact, $I_{\cap }^{\mathrm{MMI}}$ is the largest measure that is compatible with the monotonicity of mutual information (Assumption I in Section 4.1).

4.5. Further Generalizations

We finish part I of this paper by noting that one can further generalize our approach, by considering other analogues of “set”, “set size”, and “set inclusion” beyond the ones considered in Section 4.1. Such generalizations allow one to analyze notions of information intersection and union in a wide variety of domains, including setups different from the standard one considered in the PID, and domains not based on Shannon information theory.

At a general level, consider a set of object $\mathsf{\Omega }$ that represents possible “sources”, which may be random variables, as in Section 4.1, or otherwise. Assume there is some function $\varphi :\mathsf{\Omega }\to \mathbb{R}$ that quantifies the “amount of information” in a given source $\mathsf{\Omega }$, and some relation ⊏ on $\mathsf{\Omega }$ that indicates which sources are more informative than others. Then, in analogy to Equations (5) and (6), for any set of sources $\{b_1,\dots ,b_n\}\subseteq \mathsf{\Omega }$, one can define redundancy and union information as

$$\begin{array}{cc}\hfill I_{\cap }(b_1;\dots ;b_n)& :=\underset{a\in \mathsf{\Omega }}{\sup }\varphi \left(a\right)\hspace{1em}\mathrm{such}\mathrm{that}\hspace{1em}\forall ia⊏b_i\hfill \end{array}$$ (21)

$$\begin{array}{cc}\hfill I_{\cup }(b_1;\dots ;b_n)& :=\underset{a\in \mathsf{\Omega }}{\inf }\varphi \left(a\right)\hspace{1em}\mathrm{such}\mathrm{that}\hspace{1em}\forall i{b}_i⊏a.\hfill \end{array}$$ (22)

Synergy, unique, and excluded information can then be defined via Equations (1) to (3).

There are many possible examples of such generalizations, of which we mention a few as illustrations.

• Shannon information theory (beyond mutual information). In Section 4.1, $\varphi$ was the mutual information between each random variable and some target Y. This can be generalized by choosing a different “amount of information” function $\varphi$, so that redundancy and union information are quantified in terms of other measures of statistical dependence. Among many other options, possible choices of $\varphi$ include Pearson’s correlation (for continuous random variables) and measures of statistical dependency based f-divergences [52], Bregman divergences [53], and Fisher information [54].

• Shannon information theory (without a fixed target). The PID can also be defined for a different setup than the typical one considered in the literature. For example, consider a situation where the sources are channels ${\kappa }_{X_1|Y},\dots ,{\kappa }_{X_n|Y}$, while the marginal distribution over the target Y is left unspecified. Here one may take $\mathsf{\Omega }$ as the set of channels, $\varphi$ as the channel capacity $\varphi \left({\kappa }_{A|Y}\right):={\max }_{P_Y}{I}_{P_Y{\kappa }_{A|Y}}(A;Y)$, and ⊏ as some ordering relation on channels [24]

• Algorithmic information theory. The PID can be defined for other notions of information, such as the ones used in Algorithmic Information Theory (AIT) [55]. In AIT, “information” is not defined in terms of statistical uncertainty, but rather in terms of the program length necessary to generate strings. For example, one may take $\mathsf{\Omega }$ as the set of finite strings, ⊏ as algorithmic conditional independence ($a⊏b\mathrm{iff}K\left(y\right|b)-K(y|b,a)\le \mathrm{const}$, where $K(·|·)$ is conditional Kolmogorov complexity), and $\varphi \left(a\right):=K\left(y\right)-K\left(y\right|a)$ as the “algorithmic mutual information” with some target string y. (This setup is closely related to the notion of algorithmic “common information” [47]).

•   Quantum information theory. As a final example, the PID can be defined in the context of quantum information theory. For example, one may take $\mathsf{\Omega }$ as the set of quantum channels, ⊏ as quantum Blackwell order [56,57,58], and $\varphi \left(\mathsf{\Phi }\right)=\mathcal{I}(\rho ,\mathsf{\Phi })$, where $\mathcal{I}$ is the Ohya mutual information for some target density matrix $\rho$ under channel $\mathsf{\Phi }\in \mathsf{\Omega }$ [59].

5. Part II: Blackwell Redundancy and Union Information

In the first part of this paper, we proposed a general framework for defining PID terms. In this section, which forms part II of this paper, we develop a concrete definition of redundancy and union information by combining our general framework with a particular ordering relation ⊏. This ordering relation is called the “Blackwell order”, and it plays a fundamental role in statistics and decision theory [28,45,60]. We first introduce the Blackwell order, then use it to define measures of redundancy and union information, and finally discuss various properties of our measures.

5.1. The Blackwell Order

We begin by introducing the ordering relation that we use to define our PID. Given three random variables $B,C$ and Y, the ordering relation $B{\prec }_{Y}C$ is defined as follows:

$$\begin{array}{cc}\hfill B{\prec }_{Y}C\hspace{1em}\mathrm{iff}\hspace{1em}{P}_{B|Y}\left(b\right|y)& =\sum _c{\kappa }_{B|C}\left(b\right|c)P_{C|Y}\left(c\right|y)\hspace{1em}\mathrm{for}\mathrm{some}\mathrm{channel}{\kappa }_{B|C}\mathrm{and}\mathrm{all}b,y.\hfill \end{array}$$ (23)

We refer to the relation ${\prec }_Y$ as the Blackwell order relative to random variable Y. (Note that the Blackwell order and Blackwell’s Theorem are usually formulated in terms of channels—that is, conditional distributions like ${\kappa }_{B|Y}$ and ${\kappa }_{C|Y}$—rather than of random variables as done here. However, these two formulations are equivalent, as shown in [45]).

In words, Equation (23) means the conditional distribution by $P_{B|Y}$ can be generated by first sampling from the conditional distribution $P_{C|Y}$, and then applying some channel ${\kappa }_{B|C}$ to the outcome. The relation $B{\prec }_{Y}C$ implies that $P_{B|Y}$ is more noisy than $P_{C|Y}$ and, by the “data processing inequality” [61], B must have less mutual information about Y than C:

$$\begin{array}{cc}\hfill B& {\prec }_{Y}C\Rightarrow I(B;Y)\le I(C;Y).\hfill \end{array}$$ (24)

Intuition suggests that when $B{\prec }_{Y}C$, the information that B provides about Y is contained in the information that C provides about Y. This intuition is formalized within a decision-theoretic framework using the so-called Blackwell’s Theorem [28,45,60]. To introduce this theorem, imagine a scenario in which Y represents the state of the environment. Imagine also that there is an agent who acquires information about the environment via the conditional distribution $P_{B|Y}\left(b\right|y)$, and then uses outcome $B=b$ to select actions $a\in \mathcal{A}$ according to some “decision rule” given by the channel ${\kappa }_{A|B}$. Finally, the agent gains utility according to some utility function $u(a,y)$, which depends on the agent’s action a and the environment’s state y. The maximum expected utility achievable by any decision rule is given by

$$V_Y^{\max }(B,u):=\underset{{\kappa }_{A|B}}{\max }\sum _{y,b,a}{P}_Y\left(y\right)P_{B|Y}\left(b\right|y){\kappa }_{A|B}\left(a\right|b)u(a,y).$$ (25)

From an operational perspective, it is natural to say that B is less informative than C about Y if there is no utility function such that an agent with access to B can achieve higher expected utility than an agent with access to C. Blackwell’s Theorem states that this is precisely the case if and only if $B{\prec }_{Y}C$ [28,45]:

$$B{\prec }_{Y}C\hspace{1em}\mathrm{iff}\hspace{1em}{V}_Y^{\max }(B,u)\le V_Y^{\max }(C,u)\hspace{1em}\mathrm{for}\mathrm{all}u.$$ (26)

In some sense, this operational description of the relation ${\prec }_Y$ is deeper than the data processing inequality, Equation (24), which says that $B{\prec }_{Y}C$ is sufficient (but not necessary) for $I(B;Y)\le I(C;Y)$. In fact, it can happen that $I(B;Y)\le I(C;Y)$ even though $B{\nprec }_{Y}C$ [26,60,62].

A connection between PID and Blackwell’s theorem was first proposed in [13], which argued that the PID should be defined in an operational manner (see Section 5.3 for further discussion of [13]).

5.2. Blackwell Redundancy

We now define a measure of redundancy based on the Blackwell order. Specifically, we use our general definition of redundancy, Equation (7), while using the Blackwell order relative to Y as the “more informative” relation ⊏:

$$I_{\cap }^{\prec }(X_1;\dots ;X_n\to Y):=\underset{Q}{\sup }I(Q;Y)\hspace{1em}\mathrm{such}\mathrm{that}\hspace{1em}\forall iQ{\prec }_Y{X}_i.$$ (27)

We refer to this measure as Blackwell redundancy.

Given Blackwell’s Theorem, $I_{\cap }^{\prec }$ has a simple operational interpretation. Imagine two agents, Alice and Bob, who can acquire information about Y via different random variables, and then use this information to maximize their expected utility. Suppose that Alice has access to one of the sources $X_i$. Then, the Blackwell redundancy $I_{\cap }^{\prec }$ is the maximum information that Bob can have about Y without being able to do better than Alice on any utility function, regardless of which source Alice has access to.

Blackwell redundancy can also be used to define a measure of Blackwell unique information, $U^{\prec }(X_i\to Y|X_1;\dots ;X_n):=I(Y;X_i)-I_{\cap }^{\prec }(X_1;\dots ;X_n\to Y)$, via Equation (2). As we show in Appendix I, $U^{\prec }$ satisfies the following property, which we term the Multivariate Blackwell property.

Theorem 3. 

$U^{\prec }(X_i\to Y|X_1;\dots ;X_n)=0$ if and only if $X_i{\prec }_Y{X}_j$ for all $j\ne i$.

Operationally, Theorem 3 means that source $X_i$ has non-zero unique information iff there exists a utility function such that an agent with access to source $X_i$ can achieve higher utility than an agent with access to any other source $X_j$.

Computing $I_{\cap }^{\prec }$ involves maximizing a convex function subject to a set of linear constraints. These constraints define a feasible set which is a convex polytope, and the maximum must lie on one of the vertices of this polytope [63]. In Appendix C, we show how to solve this optimization problem. In particular, we use a computational geometry package to enumerate the vertices of the feasible set, and then choose the best vertex (code is available at [64]). In that appendix, we also prove that an optimal solution to Equation (27) can always be achieved by Q with cardinality $\left|\mathcal{Q}\right|=\left({\sum }_i\left|{\mathcal{X}}_i\right|\right)-n+1$. Note that the supremum in Equation (27) is always achieved. Note also that $I_{\cap }^{\prec }$ satisfies the redundancy axioms in Section 4.2.

As discussed above, solving the optimization problem in Equation (27) gives a (possibly non-unique) optimal random variable Q which specifies the content of the redundant information. As shown in Appendix C, solving Equation (27) also provides a set of channels ${\kappa }_{Q|X_i}$ for each source $X_i$, which identify the redundant information in each source.

Note that the Blackwell order satisfies assumptions I-III in Section 4.1, thus Blackwell redundancy satisfies the bounds derived in that section. Finally, note that like many other redundancy measures, Blackwell redundancy becomes equivalent to the measure $I_{\cap }^{\mathrm{MMI}}$ (as defined in Equation (19)) when applied to Gaussian random variables (for details, see Appendix E).

5.3. Blackwell Union Information

We now define a measure of union information using our general definition in Equation (8), while using the Blackwell order relative to Y as the “more informative” relation:

$$I_{\cup }^{\prec }(X_1;\dots ;X_n\to Y):=\underset{Q}{\inf }I(Q;Y)\hspace{1em}\mathrm{such}\mathrm{that}\hspace{1em}\forall i{X}_i{\prec }_{Y}Q.$$ (28)

We refer to this measure as Blackwell union information.

As for Blackwell redundancy, Blackwell union information can be understood in operational terms. Consider two agents, Alice and Bob, whose use information about Y to maximize their expected utility. Suppose that Alice has access to one of the sources $X_i$. Then, the Blackwell union information $I_{\cup }^{\prec }$ is the minimum information that Bob must have about Y in order to do better than Alice on any utility function, regardless of which source Alice has access to.

Blackwell union information can be used to define measures of synergy and excluded information via Equations (1) and (3). The resulting measure of excluded information $E^{\prec }(X_i\to Y|X_1;\dots ;X_n):=I_{\cup }^{\prec }(X_1;\dots ;X_n\to Y)-I(Y;X_i)$ satisfies the following property, which is the “dual” of the Multivariate Blackwell property considered in Theorem 3. (See Appendix I for the proof).

Theorem 4. 

$E^{\prec }(X_i\to Y|X_1;\dots ;X_n)=0$ if and only if $X_j{\prec }_Y{X}_i$ for all $j\ne i$.

Operationally, Theorem 4 means that there is excluded information for source $X_i$ iff there exists a utility function such that an agent with access to one of the other sources $X_j$ can achieve higher expected utility than an agent with access to $X_i$.

We discuss the problem of numerically solving the optimization problem in Equation (28) in the next subsection.

5.4. Relation to Prior Work

Our measure of Blackwell redundancy $I_{\cap }^{\prec }$ is new to the PID literature. The most similar existing redundancy measure is $I_{\cap }^{\mathrm{GH}}$ [18], which is discussed above in Section 4.4. $I_{\cap }^{\mathrm{GH}}$ is a special case of Equation (7), once the “more informative” relation $B⊏C$ is defined in terms of conditional independence $B-C-Y$. Note that conditional independence is stronger than the Blackwell order: given the definition of ${\prec }_Y$ in Equation (23), it is clear that $B-C-Y$ implies $B{\prec }_{Y}C$ (the channel ${\kappa }_{B|C}$ can be taken to be $P_{B|C}$), but not vice versa. As discussed in Section 4.1, stronger ordering relations give smaller values of redundancy, so in general $I_{\cap }^{\mathrm{GH}}\le I_{\cap }^{\prec }$. Note also that $B{\prec }_{Y}C$ depends only on the pairwise marginals $P_{BY}$ and $P_{CY}$, while conditional independence $B-C-Y$ depends on the joint distribution $P_{BCY}$. As we discuss in Appendix F, the conditional independence order can be interpreted in decision-theoretic terms, which suggests an operational interpretation for $I_{\cap }^{\mathrm{GH}}$.

Interestingly, Blackwell union information $I_{\cup }^{\prec }$ is equivalent to two measures that have been previously proposed in the PID literature, although they were formulated in a different way. Bertschinger et al. [13] considered the following measure of bivariate redundancy:

$$I_{\cap }^{\mathrm{BROJA}}(X_1;X_2\to Y):=I(Y;X_1)+I(Y;X_2)-I_{\cup }^{\mathrm{BROJA}}(X_1;X_2\to Y),$$ (29)

where $I_{\cup }^{\mathrm{BROJA}}$ is defined via the optimization problem

$$I_{\cup }^{\mathrm{BROJA}}(X_1;X_2\to Y)=\underset{{\tilde{X}}_1,{\tilde{X}}_2}{\min }I(Y;{\tilde{X}}_1,{\tilde{X}}_2)\hspace{1em}\mathrm{such}\mathrm{that}\hspace{1em}{P}_{{\tilde{X}}_{1}Y}=P_{X_{1}Y},P_{{\tilde{X}}_{2}Y}=P_{X_{2}Y},$$ (30)

and reflects the minimal mutual information that two random variables can have about Y, given that their pairwise marginals with Y are fixed to be $P_{X_{1}Y}$ and $P_{X_{2}Y}$. Note that Ref. [13] did not refer to $I_{\cup }^{\mathrm{BROJA}}$ as a measure of union information (we use our notation in writing it as $I_{\cup }^{\mathrm{BROJA}}$). Instead, these measures were derived from an operational motivation, with the goal of deriving a unique information measure that obeys the so-called Blackwell property: $I(Y;X_1)-I_{\cap }^{\mathrm{BROJA}}(X_1;X_2\to Y)=0$ if $X_1{\prec }_Y{X}_2$ (see Theorems 3 and 4 above).

Starting from a different motivation, Griffith and Koch [17] proposed a multivariate version of $I_{\cup }^{\mathrm{BROJA}}$,

$$I_{\cup }^{\mathrm{BROJA}}(X_1;\dots ;X_n\to Y)=\underset{{\tilde{X}}_i,\dots ,{\tilde{X}}_n}{\min }I(Y;{\tilde{X}}_1,\dots ,{\tilde{X}}_n)\hspace{1em}\mathrm{such}\mathrm{that}\hspace{1em}\forall i{P}_{{\tilde{X}}_{i}Y}=P_{X_{i}Y}.$$ (31)

The goal of Ref. [17] was to derive a measure of multivariate synergy from a measure of union information, as in Equation (1). In that paper, $I_{\cup }^{\mathrm{BROJA}}$ was explicitly defined as a measure of union information. To our knowledge, Ref. [17] was the first (and perhaps only) paper to propose a measure of union information that was not derived from redundancy via the inclusion-exclusion principle.

While $I_{\cup }^{\mathrm{BROJA}}(X_1;\dots ;X_n\to Y)$ and $I_{\cup }^{\prec }(X_1;\dots ;X_n\to Y)$ are stated as different optimization problems, we prove in Appendix G that these optimization problems are equivalent, in that they will always achieve the same optimum value. Interestingly, since $I_{\cup }^{\mathrm{BROJA}}$ and $I_{\cup }^{\prec }$ are equivalent, our measure of Blackwell redundancy $I_{\cap }^{\prec }$ appears as the natural dual to $I_{\cup }^{\mathrm{BROJA}}$. Another implication of this equivalence is that Blackwell union information $I_{\cup }^{\prec }$ can be quantified by solving the optimization problem in Equation (31), rather than Equation (28). This is advantageous, because Equation (31) involves the minimization of a convex function over a convex polytope, which can be solved using standard convex optimization techniques [65].

In Ref. [13], the redundancy measure $I_{\cap }^{\mathrm{BROJA}}$ in Equation (29) was only defined for the bivariate case. Since then, it has been unclear how to extend this redundancy measure to more than two sources. However, by comparing Equations (14) and (29), we see the root of the problem: $I_{\cap }^{\mathrm{BROJA}}$ is derived by applying the inclusion-exclusion principle to a measure of union information, $I_{\cup }^{\mathrm{BROJA}}$. It cannot be extended to more than two sources because the inclusion-exclusion principle generally leads to counterintuitive results for more than 2 sources, as shown in Lemma 1. Note also that what Ref. [13] called the unique information in $X_1$, $I_{\cup }^{\mathrm{BROJA}}(X_1;X_2\to Y)-I(Y;X_2)$, in our framework would be considered a measure of the excluded information for $X_2$.

At the same time, the union information measure $I_{\cup }^{\mathrm{BROJA}}$, and the corresponding synergy from Equation (1), does not use the inclusion-exclusion principle. Therefore, it can be easily extended to any number of sources [17].

5.5. Continuity of Blackwell Redundancy and Union Information

It is often desired that information-theoretic measures are continuous, meaning that small changes in underlying probability distributions lead to small changes in the resulting measures. In this section, we consider the continuity of our proposed measures, $I_{\cap }^{\prec }$ and $I_{\cup }^{\prec }$.

We first consider Blackwell redundancy $I_{\cap }^{\prec }$. It turns out that this measure is not always continuous in the joint probability $P_{X_1\dots X_{n}Y}$ (a discontinuous example is provided in Section 5.6). However, the discontinuity of $I_{\cap }^{\prec }$ is not necessarily pathological, and we can derive an interpretable geometric condition that guarantees that $I_{\cap }^{\prec }$ is continuous.

Consider the conditional distribution of the target Y given some source $X_i$, $P_{Y|X_i}$. Let $\mathrm{rank}{P}_{Y|X_i}$ indicate its rank, meaning the dimension of the space spanned by the vectors ${\left\{P_{Y|X_i=x_i}\right\}}_{x_i\in {\mathcal{X}}_i}$. The rank of $P_{Y|X_i}$ quantifies the number of independent directions that the target distribution $P_Y$ can be moved by manipulating the source distribution $P_{X_i}$, and it cannot be larger than $\left|\mathcal{Y}\right|$. The next theorem shows that $I_{\cap }^{\prec }$ is locally continuous, as long as $n-1$ or more of the source conditional distributions have this maximal rank.

Theorem 5. 

As a function of the joint distribution $P_{X_1,\dots ,X_n,Y}$, $I_{\cap }^{\prec }$ is locally continuous whenever $n-1$ or more of the conditional distributions $P_{Y|X_i}$ have $\mathrm{rank}{P}_{Y|X_i}=\left|\mathcal{Y}\right|$.

In proving this result, we also show that $I_{\cap }^{\prec }$ is continuous almost everywhere (see proof in Appendix D). Finally, in that appendix we also use Theorem 5 to show that $I_{\cap }^{\prec }$ is continuous everywhere if Y is a binary random variable.

We illustrate the meaning of Theorem 5 visually in Figure 2. We show two situations, both of which involve two sources $X_1$ and $X_2$ and a target Y with cardinality $\left|\mathcal{Y}\right|=3$. In one situation, both pairwise conditional distributions have rank equal to $\left|\mathcal{Y}\right|$, so $I_{\cap }^{\prec }$ is locally continuous. In the other situation, both pairwise conditional distributions are rank deficient (e.g., this might happen because $X_1$ and $X_2$ have cardinality $|{\mathcal{X}}_1|=|{\mathcal{X}}_2|=2$), so $I_{\cap }^{\prec }$ is not guaranteed to be continuous. From the figure it is easy to see how the discontinuity may arise. Given the definition of the Blackwell order and $I_{\cap }^{\prec }$, for any random variable Q in the feasible set of Equation (27), the conditional distributions $P_{Y|Q=q}$ must fall within the intersection of the distributions spanned by $P_{Y|X_1}$ and $P_{Y|X_2}$ (the intersection of the red and green shaded regions in Figure 2). On the right, the size of this intersection can discontinuously jump from a line (when $P_{Y|X_1}$ and $P_{Y|X_2}$ are perfectly aligned) to a point (when $P_{Y|X_1}$ and $P_{Y|X_2}$ are not perfectly aligned). Thus, the discontinuity of $I_{\cap }^{\prec }$ arises from a geometric phenomenon, which is related to the discontinuity of the intersection of low-dimensional vector subspaces.

Figure 2.

Illustration of Theorem 5, which provides a sufficient condition for the local continuity of $I_{\cap }^{\prec }$. Consider two scenarios, both of which involves two sources $X_1$ and $X_2$ and a target Y with cardinality $\left|\mathcal{Y}\right|=3$. The blue areas indicate the simplex of probability distributions over $\mathcal{Y}$, with the marginal $P_Y$ and the pairwise conditionals $P_{Y|X_i=x_i}$ marked. On the left, both sources have $\mathrm{rank}{P}_{Y|X_i}=3=\left|\mathcal{Y}\right|$, so $I_{\cap }^{\prec }$ is locally continuous. On the right, both sources have $\mathrm{rank}{P}_{Y|X_i}=2<\left|\mathcal{Y}\right|$, so $I_{\cap }^{\prec }$ is not necessarily locally continuous. Note that $I_{\cap }^{\prec }$ is also continuous if only source has $\mathrm{rank}{P}_{Y|X_i}=3$.

We briefly comment on the continuity of $I_{\cup }^{\prec }$. As we described above, this measure turns out to be equivalent to $I_{\cup }^{\mathrm{BROJA}}$. The continuity of $I_{\cup }^{\mathrm{BROJA}}$ in the bivariate case was proven in Theorem 35 of Ref. [66]. We believe that the continuity of $I_{\cup }^{\mathrm{BROJA}}$ for an arbitrary number of sources can be shown using similar methods, although we leave this for future work.

5.6. Behavior on the COPY Gate

As mentioned in Section 3, the “COPY gate” example is often used to test the behavior of different redundancy measures. The COPY gate has two sources, $X_1$ and $X_2$, and a target $Y=(X_1,X_2)$ which is a copy of the joint outcome. It is expected that redundancy should vanish if $X_1$ and $X_2$ are statistically independent, as formalized by the Independent identity property in Equation (4).

Blackwell redundancy $I_{\cap }^{\prec }$ satisfies the Independent identity. In fact, we prove a more general result, which shows that $I_{\cap }^{\prec }(X_1,X_2\to (X_1,X_2))$ is equal to an information-theoretic measure called Gács-Körner common information $C(X\wedge Y)$ [16,47,67]. $C(X\wedge Y)$ quantifies the amount of information that can be deterministically extracted from both random variables X or Y, and it is closely related to the “deterministic function” order ⊲ defined in Equation (16). Formally, it can be written as

$$C(X\wedge Y)=\underset{Q}{\sup }H\left(Q\right)\hspace{1em}\mathrm{such}\mathrm{that}\hspace{1em}Q⊲X,Q⊲Y,$$ (32)

where H is Shannon entropy. In Appendix I, we prove the following result.

Theorem 6. 

$I_{\cap }^{\prec }(X_1,X_2\to (X_1,X_2))=C(X_1\wedge X_2)$.

Note that $0\le C(X_1\wedge X_2)\le I(X_1;X_2)$ [47], so $I_{\cap }^{\prec }$ satisfies the Independent identity property. At the same time, $C(X_1\wedge X_2)$ can be strictly less than $I(X_1;X_2)$. For example, if $P_{X_1{X}_2}$ has full support, then $I(X_1;X_2)$ can be arbitrarily large while $C(X_1\wedge X_2)=0$ (see proof of Theorem 6). This means that $I_{\cap }^{\prec }$ violates a previously proposed property, sometimes called the Identity property, that suggests that redundancy should satisfy $I_{\cap }(X_1;X_2\to (X_1,X_2))=I(X_1;X_2)$. However, the validity of the Identity property is not clear, and several papers have argued against it [15,39].

The value of $C(X_1\wedge X_2)$ depends on the precise pattern of zeros in the joint distribution $P_{X_1{X}_2}$ and is therefore not continuous. For instance, for the bivariate COPY gate, redundancy can change discontinuously as one goes from the situation where $X_1=X_2$ (so that all information is redundant, $I_{\cap }^{\prec }=I(X_1;X_2)$) to one where $X_1$ and $X_2$ are almost, but not entirely, identical. This discontinuity can be understood in terms of Theorem 5 and Figure 2: in the COPY gate, the cardinality of the target variable $\left|\mathcal{Y}\right|=|{\mathcal{X}}_1|\times |{\mathcal{X}}_2|$ is larger than the cardinality of the individual sources. In other words, when the sources $X_1$ and $X_2$ are not perfectly correlated, they provide information about different “subspaces” of the target $(X_1,X_2)$, and so it is possible that very little (or none) of their information is redundant.

At the same time, the Blackwell property, Theorem 3, implies that

$$I_{\cap }^{\prec }(X_1,X_2\to X_1)=I(X_1;X_2)=I_{\cap }^{\prec }(X_1,X_2\to X_2)$$ (33)

In other words, the redundancy in $X_1$ and $X_2$, where either one of the individual sources is taken as the target, is given by the mutual information $I(X_1;X_2)$. This holds even though the redundancy in the COPY gate can be much lower than $I(X_1;X_2)$.

It is also interesting to consider how Blackwell union information, $I_{\cup }^{\prec }$, behaves on the COPY gate. Using techniques from [13], it can be shown that the union information is simply the joint entropy,

$$\begin{array}{cc}\hfill I_{\cup }^{\prec }(X_1;X_2\to (X_1,X_2))& =H(X_1,X_2).\hfill \end{array}$$ (34)

Since $H(X_1,X_2)=I(X_1,X_2;X_1,X_2)$, Equations (1) and (34) together imply that the COPY gate has no synergy.

Note that we can use Theorem 6 and Equation (34) to illustrate that $I_{\cap }^{\prec }$ and $I_{\cup }^{\prec }$ violate the inclusion-exclusion principle, Equation (14). Using Equation (34) and a bit of rearranging, Equation (14) becomes equivalent to $I_{\cap }^{\prec }(X_1;X_2\to (X_1,X_2))\stackrel{?}{=}I(X_1;X_2)$, which is the Identity property mentioned above. $I_{\cap }^{\prec }$ violates this property, since redundancy for the COPY gate can be smaller than $I(X_1;X_2)$.

6. Examples and Comparisons to Previous Measures

In this section, we compare our proposed measure of Blackwell redundancy $I_{\cap }^{\prec }$ to existing redundancy measures. We focus on redundancy, rather than union information, because redundancy has seen much more development in the literature, and because Blackwell union information $I_{\cup }^{\prec }$ is equivalent to an existing measure (see Section 5.4).

6.1. Qualitative Comparison

In Table 1, we compare $I_{\cap }^{\prec }$ to six existing measures of multivariate redundancy:

• $I_{\cap }^{\mathrm{WB}}$, the redundancy measure first proposed by Williams and Beer [11].

• $I_{\cap }^{\mathrm{MMI}}$, the “minimum mutual information” [51], Equation (19) in Section 4.4.

• $I_{\cap }^{⊲}$, proposed by Griffith et al. [16], Equation (17) in Section 4.4.

• $I_{\cap }^{\mathrm{GH}}$, proposed by Griffith and Ho [18], Equation (18) in Section 4.4.

• $I_{\cap }^{\mathrm{Ince}}$, proposed by Ince [20].

• $I_{\cap }^{\mathrm{FL}}$, proposed by Finn and Lizier [21].

We also compare $I_{\cap }^{\prec }$ to three existing measures of bivariate redundancy (i.e., for 2 sources):

• $I_{\cap }^{\mathrm{BROJA}}$, proposed by Bertschinger et al. [13], defined in Equation (29).

• $I_{\cap }^{\mathrm{Harder}}$, proposed by Harder et al. [19].

• $I_{\cap }^{\mathrm{dep}}$, proposed by James et al. [15].

For $I_{\cap }^{\prec }$ as well as the 9 existing measures, we consider the following properties, which are chosen to highlight differences between our approach and previous proposals:

1. Has it been defined for more than 2 sources

2. Does it obey the Monotonicity axiom from Section 4.2

3. Is it compatible with the inclusion-exclusion principle (IEP) for the bivariate case, such that union information as defined in Equation (14) obeys $I_{\cup }(X_1;X_2\to Y)\le I(X_1,X_2;Y)$

4. Does it obey the Independent identity property, Equation (4)

5. Does it obey the Blackwell property (possibly in its multivariate form, Theorem 3)

We also consider two additional properties, which require a bit of introduction.

Table 1.

Comparison of different redundancy measures. ? indicate properties that we could not easily establish.

	${\mathit{I}}_{\cap }^{\prec }$	${\mathit{I}}_{\cap }^{\mathbf{WB}}$	${\mathit{I}}_{\cap }^{\mathbf{MMI}}$	${\mathit{I}}_{\cap }^{⊲}$	${\mathit{I}}_{\cap }^{\mathbf{GH}}$	${\mathit{I}}_{\cap }^{\mathbf{Ince}}$	${\mathit{I}}_{\cap }^{\mathbf{FL}}$	${\mathit{I}}_{\cap }^{\mathbf{BROJA}}$	${\mathit{I}}_{\cap }^{\mathbf{Harder}}$	${\mathit{I}}_{\cap }^{\mathbf{dep}}$
More than 2 sources	✓	✓	✓	✓	✓	✓	✓
Monotonicity	✓	✓	✓	✓	✓			✓	✓	✓
IEP for bivariate case		✓	✓			?	?	✓	✓	✓
Independent identity	✓			✓	✓	✓		✓	✓	✓
Blackwell property	✓							✓	✓
Pairwise marginals	✓	✓	✓	✓				✓	✓
Target equality	✓	✓	✓		✓			✓	✓

The first property was suggested by Ref. [13], who argued that redundancy should only depend on the pairwise marginal distributions of each source with the target,

$$\mathrm{If}{p}_{X_{i}Y}=p_{{\tilde{X}}_i\tilde{Y}}\mathrm{for}\mathrm{all}i,\mathrm{then}{I}_{\cap }(X_1;\dots ;X_n\to Y)=I_{\cap }({\tilde{X}}_1;\dots ;{\tilde{X}}_n\to \tilde{Y}).$$ (35)

In Table 1, we term this property Pairwise marginals. We believe that the validity of Equation (35) is not universal, but may depend on the particular setting in which the PID is being used. However, redundancy redundancy measures that satisfy this property have one important advantage: they are well-defined not only when the sources are random variables $X_1,\dots ,X_n$, but also in the more general case when the sources are channels ${\kappa }_{X_1|Y},\dots ,{\kappa }_{X_n|Y}$.

The second property has not been previously considered in the literature, although it appears to be highly intuitive. Observe that the target random variable Y contains all possible information about itself. Thus, it may be expected that adding the target to the set of sources should not decrease the redundancy:

$$I_{\cap }(X_1;\dots ;X_n;Y\to Y)=I_{\cap }(X_1;\dots ;X_n\to Y).$$ (36)

In Table 1, we term this property Target equality. Note that for redundancy measures which can be put in the form of Equation (7), Target Equality is satisfied if the order ⊏ obeys $X_i⊏Y$ for all sources $X_i$. (Note also that Target Equality is unrelated to the previously proposed Strong Symmetry property; for instance, it is easy to show that the redundancy measures $I_{\cap }^{\mathrm{WB}}$ and $I_{\cap }^{\mathrm{MMI}}$ satisfy Target Equality, even though they violate Strong Symmetry [68]).

6.2. Quantitative Comparison

We now illustrate our proposed measure of redundancy $I_{\cap }^{\prec }$ on some simple examples, and compare its behavior to existing redundancy measures.

The values of $I_{\cap }^{\prec }$ were computed with our code, provided at [64]. The values of all other redundancy measures except $I_{\cap }^{\mathrm{GH}}$ were computed using the dit Python package [69]. To our knowledge, there have been no previous proposals for how to compute $I_{\cap }^{\mathrm{GH}}$. In fact, this measure involves maximizing a convex function subject to linear constraints, and can be computed using similar methods as $I_{\cap }^{\prec }$. We provide code for computing $I_{\cap }^{\mathrm{GH}}$ at [64].

We begin by considering some simple bivariate examples. In all cases, the sources $X_1$ and $X_2$ are binary and uniformly distributed. The results are shown in Table 2.

Table 2.

Behavior of $I_{\cap }^{\prec }$ and other redundancy measures on bivariate examples.

Target	${\mathit{I}}_{\cap }^{\prec }$	${\mathit{I}}_{\cap }^{\mathbf{WB}}$	${\mathit{I}}_{\cap }^{\mathbf{MMI}}$	${\mathit{I}}_{\cap }^{\wedge }$	${\mathit{I}}_{\cap }^{\mathbf{GH}}$	${\mathit{I}}_{\cap }^{\mathbf{Ince}}$	${\mathit{I}}_{\cap }^{\mathbf{FL}}$	$\begin{array}{c}{\mathit{I}}_{\cap }^{\mathbf{BROJA}}\\ {\mathit{I}}_{\cap }^{\mathbf{Harder}}\end{array}$	${\mathit{I}}_{\cap }^{\mathbf{dep}}$
$Y=X_1\mathrm{AND}{X}_2$	0.311	0.311	0.311	0	0.123	0.104	0.561	0.311	0.082
$Y=X_1+X_2$	0.5	0.5	0.5	0	0	0	0.5	0.5	0.189
$Y=X_1$	$I(X_1;X_2)$	$I(X_1;X_2)$	$I(X_1;X_2)$	$C(X_1\wedge X_2)$	$I(X_1;X_2)$	*	1	$I(X_1;X_2)$	$I(X_1;X_2)$
$Y=(X_1,X_2)$	$C(X_1\wedge X_2)$	1	1	$C(X_1\wedge X_2)$	$C(X_1\wedge X_2)$	*	1	$I(X_1;X_2)$	$I(X_1;X_2)$

1. The AND gate, $Y=X_1\mathrm{AND}{X}_2$, with $X_1$ and $X_2$ independent. (It is incorrectly stated in Refs. [18,49] that $I_{\cap }^{\mathrm{GH}}$ vanishes here; actually $I_{\cap }^{\mathrm{GH}}(X_1;X_2\to X_1\mathrm{AND}{X}_2)\approx 0.123$, which corresponds to the maximum achieved in Equation (18) by $Q=X_1\mathrm{OR}{X}_2$.)

2. The SUM gate: $Y=X_1+X_2$, with $X_1$ and $X_2$ independent.

3. The UNQ gate: $Y=X_1$. Here $I_{\cap }^{\mathrm{Ince}}$ (marked with ∗) gave values that increased with the amount of correlation between $X_1$ and $X_2$ but were typically larger than $I(X_1;X_2)$.

4. The COPY gate: $Y=(X_1,X_2)$. Here, our redundancy measure is equal to the Gács-Körner common information between X and Y, as discussed in Section 5.6. The same holds for the redundancy measures $I_{\cap }^{\mathrm{GH}}$ and $I_{\cap }^{⊲}$, which can be shown using a slight modification of the proof of Theorem 6. For this gate, $I_{\cap }^{\mathrm{Ince}}$ (marked with ∗) gave the same values as for the UNQ gate, which increased with the amount of correlation between $X_1$ and $X_2$ but were typically larger than $I(X_1;X_2)$.

We also analyze several examples with three sources, with the results shown in Table 3. We considered those previously proposed measures which can be applied to more than two sources (we do not show $I_{\cap }^{\mathrm{GH}}$, as our implementation was too slow for these examples).

Table 3.

Behavior of $I_{\cap }^{\prec }$ and other redundancy measures on three sources.

Target	${\mathit{I}}_{\cap }^{\prec }$	${\mathit{I}}_{\cap }^{\mathbf{WB}}$	${\mathit{I}}_{\cap }^{\mathbf{MMI}}$	${\mathit{I}}_{\cap }^{\wedge }$	${\mathit{I}}_{\cap }^{\mathbf{Ince}}$	${\mathit{I}}_{\cap }^{\mathbf{FL}}$
$Y=X_1\mathrm{AND}{X}_2\mathrm{AND}{X}_3$	0.138	0.138	0.138	0	0.024	0.294
$Y=X_1+X_2+X_3$	0.311	0.311	0.311	0	0	0.561
$Y=\left(\right(A,B),(A,C),(A,D\left)\right)$	1	2	2	1	1	2

1. Three-way AND gate: $Y=X_1\mathrm{AND}{X}_2\mathrm{AND}{X}_3$, where the sources are binary and uniformly and independently distributed.

2. Three-way SUM gate: $Y=X_1+X_2+X_3$, where the sources are binary and uniformly and independently distributed.

3. “Overlap” gate: we defined four independent uniformly distributed binary random variables, $A,B,C,D$. These were grouped into three sources $X_1,X_2,X_3$ as $X_1=(A,B)$, $X_2=(A,C)$, $X_3=(A,D)$. The target was the joint outcome of all three sources, $Y=(X_1,X_2,X_3)=((A,B),(A,C),(A,D))$. Note that the three sources overlap on a single random variable A, which suggests that the redundancy should be 1 bit.

7. Discussion and Future Work

In this paper, we proposed a new general framework for defining the partial information decomposition (PID). Our framework was motivated in several ways, including a formal analogy with intersections and unions in set theory as well as an axiomatic derivation.

We also used our general framework to propose concrete measures of redundancy and union information, which have clear operational interpretations based on Blackwell’s theorem. Other PID measures, such as synergy and unique information, can be computed from our measures of redundancy and union information via simple expressions.

One unusual aspect of our framework is that it provides separate measures of redundancy and union information. As we discuss above, most prior work on the PID assumed that redundancy and union information are related to each other via the so-called “inclusion-exclusion” principle. We argue that the inclusion-exclusion principle should not be expected to hold in the context of the PID, and in fact that it leads to counterintuitive behavior once 3 or more sources are present. This suggests that different information decompositions should be derived for redundancy vs. union information. This idea is related to a recent proposal in the literature, which argues that two different PIDs are needed, one based on redundancy and one based on synergy [41]. An interesting direction for future work is to relate our framework with the dual decompositions proposed in [41].

From a practical standpoint, an important direction for future work is to develop better schemes for computing our redundancy measure. This measure is defined in terms of a convex maximization problem, which in principle can be NP-hard (a similar convex maximization problem was proven to be NP-hard in [70]). Our current implementation, which enumerates the vertices of the feasible set, works well for relatively small state spaces, but we do not expect it to scale to situations with many sources, or where the sources have large cardinalities. However, the problem of convex maximization with linear constraints is a very active area of optimization research, with many proposed algorithms [63,71,72]. Investigating these algorithms, as well as various approximation schemes such as relaxations and variational bounds, is of interest.

Finally, we showed how our framework can be used to define measures of redundancy and union information in situations that go beyond the standard setting of the PID (e.g., when the probability distribution of the target is not specified). Our framework can even be applied in domains beyond Shannon information theory, such as algorithmic information theory and quantum information theory. Future work may exploit this flexibility to explore various new applications of the PID.

Appendix A. PID Axioms

In developing the PID framework, Williams and Beer [11,12] proposed that any measure of redundancy should obey a set of axioms. In slightly modified form, these axioms can be written as follows:

• Symmetry: $I_{\cap }(X_1;\dots ;X_n\to Y)$ is invariant to the permutation of $X_1,\dots ,X_n$.

• Self-redundancy: $I_{\cap }(X_1\to Y)=I(Y;X_1)$.

• Monotonicity: $I_{\cap }(X_1;\dots ;X_n\to Y)\le I_{\cap }(X_1;\dots ;X_{n-1}\to Y)$.

• Deterministic equality:$I_{\cap }(X_1;\dots ;X_n\to Y)=I_{\cap }(X_1;\dots ;X_{n-1}\to Y)$ if $X_i=f\left(X_n\right)$ for some $i<n$ and deterministic function f.

These axioms are based on intuitions regarding the behavior of intersection in set theory [12]. The Symmetry axiom is self-explanatory. Self-redundancy states that if only a single-source is present, all of its information is redundant. Monotonicity states that redundancy should not increase when an additional source is considered (consider that the size of set intersection can only decrease as more sets are considered). Deterministic equality states that redundancy should remain the same when an additional source $X_n$ is added that contains all (or more) of the same information that is already contained in an existing source $X_i$ (which is formalized as the condition $X_i=f\left(X_n\right)$).

Union information was considered the original PID proposal [12,73], as well as a more recent paper [17]. Ref. [17] proposed that any measure of union information should satisfy the following set of natural axioms, stated here in slightly modified form:

• Symmetry: $I_{\cup }(X_1;\dots ;X_n\to Y)$ is invariant to the permutation of $X_1,\dots ,X_n$.

• Self-union: $I_{\cup }(X_1\to Y)=I(Y;X_1)$.

• Monotonicity: $I_{\cup }(X_1;\dots ;X_n\to Y)\ge I_{\cup }(X_1;\dots ;X_{n-1}\to Y)$.

• Deterministic equality: $I_{\cup }(X_1;\dots ;X_n\to Y)=I_{\cup }(X_1;\dots ;X_{n-1}\to Y)$ if $X_n=f\left(X_i\right)$ for some $i<n$ and deterministic function f.

These axioms are based on intuitions concerning the behavior of the union operator in set theory, and are the natural “duals” of the redundancy axioms mentioned above.

Appendix B. Uniqueness Proofs

Proof of Theorem 1. 

Assume there is a redundancy measure $I_{\cap }^{\prime }$ that obeys the five axioms stated in the theorem. We will show that $I_{\cap }^{\prime }=I_{\cap }$, as defined in Equation (7).

Given Equation (7) and the definition of the supremum, for any $ϵ>0$ there exists a random variable Q such that $Q⊏X_i$ for $i\in \{1,\dots ,n\}$ and

$$I(Q;Y)\ge I_{\cap }(X_1;\dots ;X_n\to Y)-ϵ,$$ (A1)

By Order equality, $I_{\cap }^{\prime }(Q;X_1;\dots ;X_k\to Y)=I_{\cap }^{\prime }(Q;X_1;\dots ;X_{k-1}\to Y)$. Induction gives

$$I_{\cap }^{\prime }(Q;X_1;\dots ;X_n\to Y)=I_{\cap }^{\prime }(Q\to Y)=I(Q;Y)\ge I_{\cap }(X_1;\dots ;X_n\to Y)-ϵ$$

where we used Self-redundancy and Equation (A1). We also have $I_{\cap }^{\prime }(Q;X_1;\dots ;X_n\to Y)\le I_{\cap }^{\prime }(X_1;\dots ;X_n\to Y)$ by Symmetry and Monotonicity. Combining gives

$$I_{\cap }(X_1;\dots ;X_n\to Y)-ϵ\le I_{\cap }^{\prime }(X_1;\dots ;X_n\to Y).$$

We now show that $I_{\cap }$ is the largest measure that satisfies Existence. Let Q be a random variable that obeys $Q⊏X_i$ for all $i\in \{1,\dots ,n\}$ and $I_{\cap }^{\prime }(X_1;\dots ;X_n\to Y)=I(Y;Q)$. Since Q falls within the feasible set of the optimization problem in Equation (7),

$$I_{\cap }^{\prime }(X_1;\dots ;X_n\to Y)=I(Q;Y)\le I_{\cap }(X_1;\dots ;X_n\to Y).$$

Combining gives

$$I_{\cap }^{\prime }(X_1;\dots ;X_n\to Y)-ϵ\le I_{\cap }(X_1;\dots ;X_n\to Y)-ϵ\le I_{\cap }^{\prime }(X_1;\dots ;X_n\to Y).$$

Since this holds for all $ϵ>0$, taking the limit $ϵ\to 0$ gives $I_{\cap }^{\prime }=I_{\cap }$. □

Proof of Theorem 2. 

Assume there is a union information measure $I_{\cup }^{\prime }$ that obeys the five axioms stated in the theorem. We will show that $I_{\cup }^{\prime }=I_{\cup }$, as defined in Equation (8).

Given Equation (8) and the definition of the infinitum, for any $ϵ>0$ there exists a random variable Q such that $Q⊏X_i$ for $i\in \{1,\dots ,n\}$ and

$$I(Q;Y)\le I_{\cup }(X_1;\dots ;X_n\to Y)+ϵ,$$ (A2)

By Order equality, $I_{\cup }^{\prime }(Q;X_1;\dots ;X_k\to Y)=I_{\cup }^{\prime }(Q;X_1;\dots ;X_{k-1}\to Y)$. Induction gives

$$I_{\cup }^{\prime }(Q;X_1;\dots ;X_n\to Y)=I_{\cup }^{\prime }(Q\to Y)=I(Q;Y)\le I_{\cup }(X_1;\dots ;X_n\to Y)+ϵ$$

where we used Self-union and Equation (A2). We also have $I_{\cup }^{\prime }(Q;X_1;\dots ;X_n\to Y)\ge I_{\cup }^{\prime }(X_1;\dots ;X_n\to Y)$ by Symmetry and Monotonicity. Combining gives

$$I_{\cup }(X_1;\dots ;X_n\to Y)+ϵ\ge I_{\cup }^{\prime }(X_1;\dots ;X_n\to Y).$$

We now show that $I_{\cap }$ is the smallest measure that satisfies Existence. Let Q be a random variable that obeys $X_i⊏Q$ for all $i\in \{1,\dots ,n\}$ and $I_{\cup }^{\prime }(X_1;\dots ;X_n\to Y)=I(Y;Q)$. Since Q falls within the feasible set of the optimization problem in Equation (8),

$$I_{\cup }^{\prime }(X_1;\dots ;X_n\to Y)=I(Q;Y)\ge I_{\cup }(X_1;\dots ;X_n\to Y).$$

Combining gives

$$I_{\cup }^{\prime }(X_1;\dots ;X_n\to Y)\le I_{\cup }(X_1;\dots ;X_n\to Y)+ϵ\le I_{\cap }^{\prime }(X_1;\dots ;X_n\to Y)+ϵ.$$

Since this holds for all $ϵ>0$, taking the limit $ϵ\to 0$ gives $I_{\cup }^{\prime }=I_{\cup }$. □

Appendix C. Computing $I_{\cap }^{\prec }$

Here we consider the optimization problem that defines our proposed measure of redundancy, Equation (27). We first prove a bound on the required cardinality of Q.

Theorem A1. 

For optimizing Equation (27), it suffices to consider Q with cardinality $\left|\mathcal{Q}\right|=\left({\sum }_i\left|{\mathcal{X}}_i\right|\right)-n+1$.

Proof. 

Consider any random variable Q with outcome set $\mathcal{Q}$ which satisfies $Q{\prec }_Y{X}_i$ for all i. We show that whenever Q has full support on $\left|\mathcal{Q}\right|>\left({\sum }_i\left|{\mathcal{X}}_i\right|\right)-n+1$ outcomes, there is another random variable $\tilde{Q}$ which achieves $I(\tilde{Q};Y)\ge I(Q;Y)$, while satisfying $\tilde{Q}{\prec }_Y{X}_i$ for all i and having support on at most $\left({\sum }_i\left|{\mathcal{X}}_i\right|\right)-n+1$ outcomes.

To begin, let $\mathsf{\Omega }$ indicate the set of random variables over outcomes $\mathcal{Q}$, such that all $\tilde{Q}\in \mathsf{\Omega }$ satisfy:

$$\begin{array}{cccc}\hfill P_{Y|\tilde{Q}}\left(y\right|q)& =P_{Y|Q}\left(y\right|q)\hfill & \hfill \hspace{1em}& \mathrm{for}\mathrm{all}y,q\in \{q\in \mathcal{Q}:P_{\tilde{Q}}\left(q\right)>0\}\hfill \end{array}$$ (A3)

$$\begin{array}{cccc}\hfill \sum _q{P}_{\tilde{Q}}\left(q\right)P_{X_i|Q}\left(x_i\right|q)& =P_{X_i}\left(x_i\right)\hspace{1em}\hfill & \hfill \hspace{1em}& \mathrm{for}\mathrm{all}i,x_i.\hfill \end{array}$$ (A4)

Since $Q{\prec }_Y{X}_i$, by Equation (23) there exist channels ${\kappa }_{Q|X_i}\left(q\right|x_i)$ that satisfy $P_{Q|Y}\left(q\right|y)={\sum }_{x_i}{\kappa }_{Q|X_i}\left(q\right|x_i)P_{X_i|Y}\left(x_i\right|y).$ Now write the conditional distribution over $\tilde{Q}$ and Y as

$$\begin{array}{cc}\hfill P_{\tilde{Q}Y}(q,y)=\frac{P_{\tilde{Q}}\left(q\right)}{P_Q\left(q\right)}{P}_{QY}(q,y)& =\sum _{x_i}\frac{P_{\tilde{Q}}\left(q\right)}{P_Q\left(q\right)}{\kappa }_{Q|X_i}\left(q\right|x_i)P_{X_i|Y}\left(x_i\right|y)P_Y\left(y\right)\hfill \\ \hfill & =\sum _{x_i}{\kappa }_{\tilde{Q}|X_i}^{\prime }\left(q\right|x_i)P_{X_i|Y}\left(x_i\right|y)P_Y\left(y\right),\hfill \end{array}$$ (A5)

where we used Equation (A3) and defined the channel ${\kappa }_{\tilde{Q}|X_i}^{\prime }$ as

$${\kappa }_{\tilde{Q}|X_i}^{\prime }=\frac{P_{\tilde{Q}}\left(q\right)}{P_{X_i}\left(x_i\right)}\left[\frac{P_{X_i}\left(x_i\right)}{P_Q\left(q\right)}{\kappa }_{Q|X_i}\left(q\right|x_i)\right],$$

(Note this is a kind of double Bayesian inverse, given Equation (A4)). Equation (A5) implies that $\tilde{Q}{\prec }_Y{X}_i$ for all i.

We now show that there is $\tilde{Q}\in \mathsf{\Omega }$ that achieves $I(Q;Y)\le I(\tilde{Q};Y)$ and has support on at most $\left({\sum }_i\left|{\mathcal{X}}_i\right|\right)-n+1$ outcomes in $\mathcal{Q}$. Write the mutual information between any $\tilde{Q}\in \mathsf{\Omega }$ and Y as

$$I(\tilde{Q};Y)=\sum _q{P}_{\tilde{Q}}\left(q\right)D_{\mathrm{KL}}(P_{Y|\tilde{Q}=q}\parallel P_Y)=\sum _q{P}_{\tilde{Q}}\left(q\right)D_{\mathrm{KL}}(P_{Y|Q=q}\parallel P_Y),$$ (A6)

where $D_{\mathrm{KL}}$ is the Kullback-Leibler divergence. We consider the maximum of this mutual information across $\mathsf{\Omega }$, $I^{*}={\max }_{\tilde{Q}\in \mathsf{\Omega }}I(\tilde{Q};Y)$. Using Equations (A4) and (A6), this maximum can be written as

$$\begin{array}{cc}\hfill I^{*}& =\underset{\omega \in \Delta }{\max }\sum _q\omega \left(q\right)D(P_{Y|Q=q}\parallel P_Y)\hspace{1em}\mathrm{such}\mathrm{that}\hspace{1em}\forall i,x_i:\sum _q\omega \left(q\right)P_{X_i|Q}\left(x_i\right|q)=P_{X_i}\left(x_i\right),\hfill \end{array}$$

where $\Delta$ is the set of all distributions over $\mathcal{Q}$. By conservation of probability, ${\sum }_{x_i}{P}_{X_i}\left(x_i\right)=1$, so we can eliminate a constraint for one of the outcomes $x_i$ of each source i. Thus, $I^{*}$ is the maximum of a linear function over $\Delta$, subject to ${\sum }_i(\left|{\mathcal{X}}_i\right|-1)=\left({\sum }_i\left|{\mathcal{X}}_i\right|\right)-n$ hyperplane constraints.

The feasible set is compact, and the maximum will be achieved at one of the extreme points of the feasible set. By Dubin’s Theorem [74], any extreme point of this feasible set can be expressed as a convex combination of at most $\left({\sum }_i\left|{\mathcal{X}}_i\right|\right)-n+1$ extreme points of $\Delta$. In other words, the maximum in Equation (27) is achieved by a random variable $\tilde{Q}$ with support on at most $\left({\sum }_i\left|{\mathcal{X}}_i\right|\right)-n+1$ values of $\mathcal{Q}$. This random variable satisfies

$$I(\tilde{Q};Y)=I^{*}\ge I(Q;Y),$$

where the last inequality comes from the fact that Q is an element of $\mathsf{\Omega }$. □

We now return to the optimization problem in Equation (27). Given Theorem A1 and the definition of the Blackwell order in Equation (23), it can be rewritten as

$$\begin{array}{cc}\hfill I_{\cap }^{\prec }(X_1;\dots ;X_n\to Y)=& \underset{{\kappa }_{Q|Y},{\kappa }_{Q|X_1,\dots ,}{\kappa }_{Q|X_n}}{\max }{I}_{\kappa }(Q,Y)\hfill \\ \hfill & \mathrm{such}\mathrm{that}\forall i,y,x_i:\sum _{x_i}{\kappa }_{Q|X_i}\left(q\right|x_i)P_{X_i|Y}\left(x_i\right|y)={\kappa }_{Q|Y}\left(q\right|y).\hfill \end{array}$$ (A7)

where the optimization is over channels with $\mathcal{Q}$ of cardinality $\left({\sum }_i\left|{\mathcal{X}}_i\right|\right)-n+1$. The notation $I_{\kappa }(Q;Y)$ indicates the mutual information that arises from the marginal distribution $P_Y$ and the conditional distribution ${\kappa }_{Q|Y}$,

$$I_{\kappa }(Q;Y)=\sum _y{P}_Y\left(y\right){\kappa }_{Q|Y}\left(q\right|y)ln\frac{{\kappa }_{Q|Y}\left(q\right|y)}{{\sum }_{y^{\prime }}{\kappa }_{Q|Y}\left(q\right|y^{\prime })P_Y\left(y^{\prime }\right)}$$

Equation (A7) involves maximizing a convex function over the convex polytope defined by the following system of linear (in)equalities:

$$\begin{array}{c}\hfill \mathsf{\Lambda }=\left\{\right({\kappa }_{Q|Y},{\kappa }_{Q|X_1,\dots ,}{\kappa }_{Q|X_n}):\end{array}$$

$$\begin{array}{cc}\hfill \forall i,x_i,q& {\kappa }_{Q|X_i}\left(q\right|x_i)\ge 0,\hfill \end{array}$$ (A8)

$$\begin{array}{cc}\hfill \forall q,y& {\kappa }_{Q|Y}\left(q\right|y)\ge 0,\hfill \end{array}$$ (A9)

$$\begin{array}{cc}\hfill \forall y& \sum _q{\kappa }_{Q|Y}\left(q\right|y)=1,\hfill \end{array}$$ (A10)

$$\begin{array}{cc}\hfill \forall i,x_i& \sum _q{\kappa }_{Q|X_i}\left(q\right|x_i)=1,\hfill \end{array}$$ (A11)

$$\begin{array}{cc}\hfill \forall i,y,q\in \mathcal{Q}\setminus \left\{0\right\}& \left[\sum _{x_i}{\kappa }_{Q|X_i}\left(q\right|x_i)P_{X_{i}Y}(x_i,y)\right]-{\kappa }_{Q|Y}\left(q\right|y)P_Y\left(y\right)=0\},\hfill \end{array}$$ (A12)

We do not place a constraint on $q=0$ in Equation (A12) because that would be redundant with the constraints Equations (A10) and (A11). Also, note that we replaced the sup in Equation (27) with max in Equation (A7), which is justified since we are optimizing over a finite dimensional, closed, and bounded region (thus the supremum is always achieved).

The maximum of a convex function over a convex polytope is found at one of the vertices of the polytope. To find the solution to Equation (A7), we use a computational geometry package to enumerate the vertices of $\mathsf{\Lambda }$. We evaluate $I_{\kappa }(Y;Q)$ at each vertex, and pick the maximum value. This procedure also finds optimal conditional distributions ${\kappa }_{Q|Y},{\kappa }_{Q|X_1,\dots ,}{\kappa }_{Q|X_n}$. Code is available at [64].

Appendix D. Continuity of $I_{\cap }^{\prec }$

To prove the continuity of $I_{\cap }^{\prec }$, we begin by considering the feasible set of the optimization problem in Equation (A7), as specified by the system (in)equalities in Equations (A8) to (A12). For convenience, write this system of (in)equalities in matrix notation,

$$\mathsf{\Lambda }=\left\{\overrightarrow{\kappa }\in {\mathbb{R}}^{\left|\mathcal{Q}\right|\left|\mathcal{Y}\right|+{\sum }_i\left|\mathcal{Q}\right||{\mathcal{X}}_i|}:\kappa \ge 0,A\kappa =a\right\},$$ (A13)

where $\overrightarrow{\kappa }$ is a vector representation of $({\kappa }_{Q|Y},{\kappa }_{Q|X_1,\dots ,}{\kappa }_{Q|X_n})$, the matrix A encodes the left-hand side of Equations (A10) to (A12), and the vector a is filled with 1s and 0s, as appropriate.

We first prove the following lemma.

Lemma A1. 

The matrix A defined in Equation (A13) is full rank if $n-1$ or more of the pairwise conditional distributions have $\mathrm{rank}{P}_{Y|X_i}=\left|\mathcal{Y}\right|$.

Proof. 

Without loss of generality, assume that $P_Y$ has full support (otherwise none of the pairwise marginals $P_{X_{i}Y}$ can achieve rank $\left|\mathcal{Y}\right|$). Write A in block matrix form as $A=\left[\begin{array}{c}B\\ C\end{array}\right]$, where the matrix B has $\left|\mathcal{Y}\right|+{\sum }_i\left|{\mathcal{X}}_i\right|$ rows and encodes the constraints of Equations (A10) and (A11), and the matrix C has $n\left|\mathcal{Y}\right|\left(\right|\mathcal{Q}|-1)$ rows and encodes the constraints of Equation (A12).

Each row in B has a 1 in some column which is zero in every other row of B and every row of C. This column corresponds either to ${\kappa }_{Q|Y}\left(0\right|y)$ for a particular y (for constraints like Equation (A10)), or to ${\kappa }_{Q|X_i}\left(0\right|x_i)$ for a particular i and $x_i$ (for constraints like Equation (A11)). These columns are 0 in C because $q=0$ is omitted Equation (A12). This means that no row of B is a linear combination of other rows in B or C, and that no row in C is a linear combination of any set of other rows that includes a row in B. Therefore, if the rows of A are linearly dependent, it must be that the rows of C are linearly dependent.

Next, let ${\overrightarrow{c}}^{i,y,q}$ indicate the row of C that represents the constraints in Equation (A12) for some source i and outcomes $y,q\ne 0$. Any such row has a column for each $x_i\in {\mathcal{X}}_i$ with value $P_{X_{i}Y}(x_i,y)$ (at the same index as the row in $\overrightarrow{\kappa }$ that represents ${\kappa }_{Q|X_i}\left(q\right|x_i)$). Since $P_{X_{i}Y}(x_i,y)>0$ for at least one $x_i\in {\mathcal{X}}_i$, one of these columns must be non-zero. At the same time, these columns are zero in every row ${\overrightarrow{c}}^{j,y,q^{\prime }}$ where $j\ne i$ or $q^{\prime }\ne q$. This means that row ${\overrightarrow{c}}^{i,y,q}$ can only be a linear combination of other rows in C if, for all $x_i$, $P_{X_{i}Y}(x_i,y)$ is a linear combination of $P_{X_{i}Y}(x_i,y^{\prime })$ for $y^{\prime }\ne y$. In linear algebra terms, this can be stated as $\mathrm{rank}{P}_{Y|X_i}<\left|\mathcal{Y}\right|$.

The previous argument shows that if A is linearly dependent, there must be at least one source i with $\mathrm{rank}{P}_{Y|X_i}<\left|\mathcal{Y}\right|$ and some row ${\overrightarrow{c}}^{i,y,q}$ which is a linear combination of other rows from C. Observe that this row ${\overrightarrow{c}}^{i,y,q}$ has a column with value $P_Y\left(y\right)>0$ (at the same index as the row in $\overrightarrow{\kappa }$ that represents ${\kappa }_{Q|Y}\left(q\right|y)$). This column is zero in every other row ${\overrightarrow{c}}^{i,y^{\prime },q^{\prime }}$ for $y^{\prime }\ne y$ or $q^{\prime }\ne q$. This means that ${\overrightarrow{c}}^{i,y,q}$ is a linear combination of a set of other rows in C that include some row ${\overrightarrow{c}}^{j,y,q}$ for $j\ne i$. This implies that ${\overrightarrow{c}}^{j,y,q}$ is also a linear combination of other rows in C, which means that $\mathrm{rank}{P}_{Y|X_j}<\left|\mathcal{Y}\right|$.

We have shown that if A is linearly dependent, there must be at least two pairwise conditionals with $\mathrm{rank}{P}_{Y|X_i}<\left|\mathcal{Y}\right|$. □

We are now ready to prove Theorem 5.

Proof of Theorem 5. 

For the case of a single source ($n=1$), $I_{\cap }^{\prec }$ reduces to the mutual information $I_{\cap }^{\prec }=I(Y;X_1)$, which is continuous (Section 2.3, [75]). Thus, without loss of generality, we assume that $n\ge 2$.

Next, we define some notation. Note that the optimum value ($I_{\cap }^{\prec }$) and the feasible set of the optimization problem in Equation (A7) is a function of the pairwise marginal distributions $P_{X_{1}Y},\dots ,P_{X_{n}Y}$. We write $\mathsf{\Omega }$ for the set of all pairwise marginal distributions which have the same marginal over Y:

$$\mathsf{\Omega }=\left\{(q_{X_{1}Y},\dots ,q_{X_{n}Y}):\sum _{x_i}{q}_{X_{i}Y}(x_i,y)=\sum _{x_j}{q}_{X_{j}Y}(x_j,y)\hspace{1em}\forall i,j\right\}.$$

For any $r\in \mathsf{\Omega }$, let $I_{\cap }^{\prec }\left(r\right)$ indicate the corresponding optimum value in Equation (A7), given the marginals in r, and let $\mathsf{\Lambda }\left(r\right)$ indicate the feasible set of the optimization problem, as defined in Equation (A13).

Note that the matrix A in Equation (A13) depends on the choice of r, which we indicate by writing it as the matrix-valued function $A\left(r\right)$. Given any $r=(q_{X_{1}Y},\dots ,q_{X_{n}Y})\in \mathsf{\Omega }$ and feasible solution $\kappa =({\kappa }_{Q|Y},{\kappa }_{Q|X_1,\dots ,}{\kappa }_{Q|X_n})\in \mathsf{\Lambda }\left(r\right)$, let $I(r,\kappa )$ indicate the corresponding mutual information $I(Q;Y)$, where the marginal distribution over Y is specified by r and the conditional distribution of Q given Y is specified by ${\kappa }_{Q|Y}$. Using this notation, $I_{\cap }^{\prec }\left(r\right)={\max }_{\kappa \in \mathsf{\Lambda }\left(r\right)}I(r,\kappa )$.

Below, we show that $I_{\cap }^{\prec }\left(r\right)$ is continuous if r is rank regular [76], which means that there is a neighborhood $U\subseteq \mathsf{\Omega }$ of r such that $\mathrm{rank}A\left(r^{\prime }\right)=\mathrm{rank}A\left(r\right)$ for all $r^{\prime }\in U$. Then, to prove the theorem, we assume that $A\left(r\right)$ is full rank. Given Lemma A1, this is true as long as $n-1$ or more of the pairwise conditionals $P_{Y|X_i}$ have $\mathrm{rank}{P}_{Y|X_i}=\left|\mathcal{Y}\right|$. Note that a matrix M is full rank iff the singular values $\sigma \left(M\right)$ are all strictly positive. Since $A\left(r\right)$ is full rank, and $A\left(r\right)$ and $\sigma \left(M\right)$ are continuous, there is a neighborhood U of r such that the singular values $\sigma \left(A\left(r^{\prime }\right)\right)$ are all strictly positive for all $r^{\prime }\in U$, therefore all $A\left(r^{\prime }\right))$ have full rank. This shows that r is rank regular and so $I_{\cap }^{\prec }$ is continuous at r.

We now prove that $I_{\cap }^{\prec }\left(r\right)$ is continuous if $A\left(r\right)$ is rank regular. To do so, we will use Hoffman’s Theorem [77,78]. In our case, it states that for any pair of marginals $r,r^{\prime }\in \mathsf{\Omega }$ and a feasible solution ${\kappa }^{\prime }\in \mathsf{\Lambda }\left(r^{\prime }\right)$, there exists a feasible solution $\kappa \in \mathsf{\Lambda }\left(r\right)$ such that

$$∥\kappa -{\kappa }^{\prime }∥\le \alpha ∥A\left(r\right)-A\left(r^{\prime }\right)∥,$$ (A14)

where $\alpha$ is a constant that does not depend on $r^{\prime }$ or ${\kappa }^{\prime }$. (In the notation of [78], we take $G=G^{\prime }$, $g=g^{\prime }$ and $d^{\prime }=d$, and use that the norm of $s={\kappa }^{\prime }$ is bounded, given that it is finite dimensional and has entries in $[0,1]$). We will also use Daniel’s theorem (Theorem 4.2, [78]), which states that for any $r,r^{\prime }\in \mathsf{\Omega }$ such that $\mathrm{rank}A\left(r\right)=\mathrm{rank}A\left(r^{\prime }\right)$, and any feasible solution $\kappa \in \mathsf{\Lambda }\left(r\right)$, there exists ${\kappa }^{\prime }\in \mathsf{\Lambda }\left(r^{\prime }\right)$ such that

$$∥\kappa -{\kappa }^{\prime }∥\le \beta ∥A\left(r\right)-A\left(r^{\prime }\right)∥,$$ (A15)

where $\beta$ is a constant that doesn’t depend on $r^{\prime }$ (in the notation of [78], ${\epsilon }^{\prime }=∥A\left(r\right)-A\left(r_i^{\prime }\right)∥$ and again use that $\kappa$ have a bounded norm).

Now consider also any sequence $r_1^{\prime },r_2^{\prime },\dots \in \mathsf{\Omega }$ that converges to a marginal $r\in \mathsf{\Omega }$. Let ${\kappa }_i^{\prime }\in \mathsf{\Lambda }\left(r_i^{\prime }\right)$ indicate an optimal solution of Equation (A7) for $r_i$, so that $I(r_i^{\prime },{\kappa }_i^{\prime })=I_{\cap }^{\prec }\left(r_i^{\prime }\right)$. Given Equation (A14), there is a corresponding sequence ${\kappa }_1,{\kappa }_2,\dots \in \mathsf{\Lambda }\left(r\right)$ such that

$$∥{\kappa }_i-{\kappa }_i^{\prime }∥\le \alpha ∥A\left(r\right)-A\left(r_i^{\prime }\right)∥.$$

Since $A(·)$ is continuous and $r_i^{\prime }$ converges to r, we have ${\lim }_{i\to \infty }A\left(r_i^{\prime }\right)=A\left(r\right)$ and therefore ${\lim }_{i\to \infty }∥{\kappa }_i-{\kappa }_i^{\prime }∥=0$. This implies

$$0=\underset{i\to \infty }{\lim }I(r_i^{\prime },{\kappa }_i^{\prime })-I(r,{\kappa }_i)\ge \underset{i\to \infty }{\lim }{I}_{\cap }^{\prec }\left(r_i^{\prime }\right)-I_{\cap }^{\prec }\left(r\right)$$ (A16)

where we first used continuity of mutual information, $I(r_i^{\prime },{\kappa }_i^{\prime })=I_{\cap }^{\prec }\left(r_i^{\prime }\right)$ and $I(r,{\kappa }_i)\le I_{\cap }^{\prec }\left(r\right)$.

Now assume that r is rank regular. Since $r_i$ converges to r, $\mathrm{rank}A\left(r_i^{\prime }\right)=\mathrm{rank}A\left(r\right)$ for all sufficiently large i. Let $\kappa \in \mathsf{\Lambda }\left(r\right)$ be an optimal solution of Equation (A7) for r, so that $I(r,\kappa )=I_{\cap }^{\prec }\left(r\right)$. Given Equation (A15), for all sufficiently large i there exists ${\kappa }_i^{\prime }\in \mathsf{\Lambda }\left(r_i^{\prime }\right)$ such that

$$∥\kappa -{\kappa }_i^{\prime }∥\le \beta ∥A\left(r\right)-A\left(r_i^{\prime }\right)∥.$$

As before, we have ${\lim }_{i\to \infty }A\left(r_i^{\prime }\right)=A\left(r\right)$ and ${\lim }_{i\to \infty }∥\kappa -{\kappa }_i^{\prime }∥=0$, which implies

$$0=\underset{i\to \infty }{\lim }I(r_i^{\prime },{\kappa }_i^{\prime })-I(r,\kappa )\le \underset{i\to \infty }{\lim }{I}_{\cap }^{\prec }\left(r_i^{\prime }\right)-I_{\cap }^{\prec }\left(r\right)$$ (A17)

where we first used continuity of mutual information, $I(r_i^{\prime },{\kappa }_i^{\prime })\le I_{\cap }^{\prec }\left(r_i^{\prime }\right)$, and $I(r,\kappa )\le I_{\cap }^{\prec }\left(r\right)$.

Combining Equations (A16) and (A17) proves continuity, ${\lim }_{i\to \infty }{I}_{\cap }^{\prec }\left(r_i\right)=I_{\cap }^{\prec }\left(r\right)$, under the assumption that $A\left(r\right)$ is rank regular.

Finally, note that $A\left(r\right)$ is a real analytic function of r. This means that almost all r rank regular, because those r which are not rank regular form a proper analytic subset of $\mathsf{\Omega }$ (which has measure zero) [76]. Thus, $I_{\cap }^{\prec }\left(r\right)$ is continuous almost everywhere. □

We finish our analysis of the continuity of $I_{\cap }^{\prec }$ by showing global continuity when the target is a binary random variable.

Corollary A1. 

$I_{\cap }^{\prec }(X_1;\dots ;X_n\to Y)$ is continuous everywhere when Y is a binary random variable.

Proof. 

In an overloading of notation, let $I_{\cap }^{\prec }\left(r\right)$ and $I_r(X_i;Y)$ indicate $I_{\cap }^{\prec }(X_1;\dots ;X_n\to Y)$ and the mutual information $I(X_i;Y)$, respectively, for the joint distribution $r_{X_1\dots X_{n}Y}$. By Theorem 5, $I_{\cap }^{\prec }$ can only be discontinuous at the joint distribution $P_{X_1\dots X_{n}Y}$ if there is a source $X_i$ with $\mathrm{rank}{P}_{Y|X_i}=1<\left|\mathcal{Y}\right|$. However, if source $X_i$ has rank 1, then the conditional distributions $P_{Y|X_i=x_i}$ are the same for all $x_i$, so $I_P(X_i;Y)=0$ and $I_{\cap }^{\prec }\left(P\right)=0$ (since $0\le I_{\cap }^{\prec }\left(P\right)\le I_P(X_i;Y)$). Finally, consider any sequence of joint distributions $s_{X_1\dots X_{n}Y}^{\left(n\right)}$ for $n=1,2,\dots$ that converges to $P_{X_1\dots X_{n}Y}$. We have

$$0\le \underset{n\to \infty }{\lim }{I}_{\cap }^{\prec }\left(s^{\left(n\right)}\right)\le \underset{n\to \infty }{\lim }{I}_{s^{\left(n\right)}}(X_i;Y)=I_P(X_i;Y)=0,$$

where we used the continuity of mutual information. This shows that ${\lim }_{n\to \infty }{I}_{\cap }^{\prec }\left(s^{\left(n\right)}\right)=0=I_{\cap }^{\prec }\left(P\right)$, proving continuity. □

Appendix E. Behavior of $I_{\cap }^{\prec }$ on Gaussian Random Variables

Although in this paper we focused on random variables with finite sets of outcomes, we can briefly comment on the behavior of Blackwell redundancy on Gaussian random variables. Suppose that all sources $X_1,\dots ,X_n$ and the target Y are continuous-valued, and that the pairwise marginals $P_{X_{i}Y}$ are multivariate Gaussians. In addition, suppose that Y is one-dimensional (the sources $X_i$ can be multi-dimensional). Given these assumptions, Barrett [51] analyzed the $I_{\cup }^{\mathrm{BROJA}}$ measure and showed that the corresponding excluded information obeys $E(X_j\to Y|X_i;X_j)=0$ whenever $I(X_i;Y)\le I(X_j;Y)$. Recall that $I_{\cup }^{\mathrm{BROJA}}$ is equivalent to Blackwell union information $I_{\cup }^{\prec }$. Then, given the Blackwell property, Theorem 4, and the data processing inequality, Equation (24), the result in Ref. [51] implies that $X_i{\prec }_Y{X}_j$ if and only if $I(X_i;Y)\le I(X_j;Y)$. Thus, for Gaussian random variables, Blackwell redundancy $I_{\cap }^{\prec }$ is equivalent to $I_{\cap }^{\mathrm{MMI}}$ redundancy, as defined in Equation (19). This parallels the case for most other redundancy measures [51].

Appendix F. Operational Interpretation of the $I_{\cap }^{\mathrm{GH}}$

As mentioned in the main text, the redundancy measure $I_{\cap }^{\mathrm{GH}}$ is a special case of Equation (7), where the “more informative” order $B⊏C$ is defined in terms of conditional independence $B-C-Y$. Here we show that this ordering relation can be given an operational interpretation, which is similar but distinct from the operational interpretation of the Blackwell order ${\prec }_Y$ discussed in Section 5.1.

To introduce this operational interpretation, let the random variable Y represent the state of the environment, and assume there are two random variables B and C which have some information about Y. Suppose that an agent tries to maximize expected utility $u(a,y)$ by using a strategy that depends either on the outcomes of B or C. Blackwell’s theorem tells us that $B{\prec }_{Y}C$ iff an agent with access to C can always achieve higher expected utility than an agent with access to B. It is possible, however, the agent with access to C may do worse than the agent with access to B, conditional on the event that random variable C has some particular outcome c. In the following theorem, we show $B-C-Y$ iff the agent cannot do better with B than C, even when conditioned on any particular outcome $C=c$. (We thank Johannes Rauh for suggesting this simplified proof).

Theorem A2. 

Given random variables B, C, and Y, $B-C-Y$ if and only if

$$\underset{{\kappa }_{A|B}}{\max }\sum _{y,a,b}{P}_{YB|C}(y,b|c){\kappa }_{A|B}\left(a\right|b)u(a,y)\le \underset{{\kappa }_{A|C}}{\max }\sum _{y,a}{P}_{Y|C}\left(y\right|c){\kappa }_{A|C}\left(a\right|c)u(a,y).$$ (A18)

for all utility functions $u(a,y)$ and all $c\in \mathcal{C}$ with $P_C\left(c\right)>0$.

Proof. 

Consider any $c\in \mathcal{C}$ with $P_C\left(c\right)>0$. By multiplying both sides of Equation (A18) by $P_C\left(c\right)$ and rearranging, this inequality can be rewritten as

$$\begin{array}{c}\underset{{\kappa }_{A|B}}{\max }\sum _{y,a,b}{P}_Y\left(y\right)P_{C|Y}\left(c\right|y)P_{B|Y,C}\left(b\right|y,c){\kappa }_{A|B}\left(a\right|b)u(a,y)\hfill \\ \hspace{1em}\hspace{1em}\hspace{1em}\le \underset{{\kappa }_{A|C}}{\max }\sum _{y,a}{P}_Y\left(y\right)P_{C|Y}\left(c\right|y){\kappa }_{A|C}\left(a\right|c)u(a,y).\hfill \end{array}$$ (A19)

Note that if Equation (A18) holds for a given c and all utility functions, then it must also hold for the utility function $u^{\prime }(a,y):=P_{C|Y}\left(c\right|y)u(a,y)$. Plugging into Equation (A19) gives

$$\begin{array}{c}\hfill \underset{{\kappa }_{A|B}}{\max }\sum _{y,a,b}{P}_Y\left(y\right)P_{B|Y,C}\left(b\right|y,c){\kappa }_{A|B}\left(a\right|b)u^{\prime }(a,y)\le \underset{{\kappa }_{A|C}}{\max }\sum _{y,a}{P}_Y\left(y\right){\kappa }_{A|C}\left(a\right|c)u^{\prime }(a,y).\end{array}$$ (A20)

Now define two random variables: a constant random variable ${\widehat{C}}_c$ with a single outcome c and ${\widehat{B}}_c$ with the same outcomes as B but having the conditional distribution $P_{{\widehat{B}}_c|Y}=P_{B|Y,C=c}$. Then, Equation (A20) can be written in terms of these random variables as

$$\underset{{\kappa }_{A|B}}{\max }\sum _{y,a,b}{P}_Y\left(y\right)P_{{\widehat{B}}_c|Y}\left(b\right|y){\kappa }_{A|B}\left(a\right|b)u^{\prime }(a,y)\le \underset{{\kappa }_{A|C}}{\max }\sum _{y,a}{P}_Y\left(y\right)P_{{\widehat{C}}_c|Y}\left(c\right|y){\kappa }_{A|C}\left(a\right|c)u^{\prime }(a,y).$$ (A21)

Given Equations (25) and (26), Equation (A21) holds for all $u^{\prime }$ iff ${\widehat{B}}_c{\prec }_Y{\widehat{C}}_c$. Since ${\widehat{C}}_c$ has a single outcome, it is independent of Y. That means ${\widehat{B}}_c$ must be also independent of Y and so $P_{{\widehat{B}}_c|Y=y}=P_{B|Y=y,C=c}$ is the same for all y, implying that $P_{B|Y=y,C=c}=P_{B|C=c}$. Since this holds for all $c\in \mathcal{C}$, $P_{B|YC}=P_{B|C}$ and therefore $B-C-Y$. □

Given Theorem A2, $I_{\cap }^{\mathrm{GH}}$ can be given the following operational interpretation. Imagine two agents, Alice and Bob, who can acquire information about Y via different random variables, and then use this information to maximize their expected utility. Suppose that Alice has access to one of the sources $X_i$. Then, $I_{\cap }^{\mathrm{GH}}$ is the maximum information that Bob can have about Y without being able to do better than Alice on any utility function, regardless of which source $X_i$ Alice has access to, and even when conditioned on $X_i$ having any particular outcome $x_i$.

Appendix G. Equivalence of $I_{\cup }^{\prec }$ and $I_{\cup }^{\mathrm{BROJA}}$

The following proves that $I_{\cup }^{\prec }$ and $I_{\cup }^{\mathrm{BROJA}}$, as defined via the optimization problems in Equations (28) and (31), are equivalent.

Theorem A3. 

$I_{\cup }^{\prec }(X_1;\dots ;X_n\to Y)=I_{\cup }^{\mathrm{BROJA}}(X_1;\dots ;X_n\to Y).$

Proof. 

Let ${\tilde{X}}_1,\dots ,{\tilde{X}}_n$ be a set of random variables that achieve $I(Y;{\tilde{X}}_1,\dots ,{\tilde{X}}_n)=I_{\cup }^{\mathrm{BROJA}}$$(X_1;\dots ;X_n\to Y)$. Define the random variable $Q:=({\tilde{X}}_1,\dots ,{\tilde{X}}_n)$, and note that ${\tilde{X}}_i{\prec }_{Y}Q$ for all i. Since $P_{{\tilde{X}}_{i}Y}=P_{X_{i}Y}$ for all i, it must be that $X_i{\prec }_{Y}Q$ for all i. Thus Q satisfies the constraints of the optimization problem in Equation (28), so

$$I_{\cup }^{\prec }(X_1;\dots ;X_n\to Y)\le I(Y;Q)=I_{\cup }^{\mathrm{BROJA}}(X_1;\dots ;X_n\to Y).$$ (A22)

Next, consider the optimization in Equation (28). For any $ϵ>0$, let Q be a random variable that satisfies $X_i{\prec }_{Y}Q$ and achieves

$$I(Y;Q)\le I_{\cup }^{\prec }(X_1;\dots ;X_n\to Y)+ϵ.$$ (A23)

For each i, let ${\kappa }_{X_i|Q}$ be a channel that obeys $P_{X_i|Y}\left(x_i\right|y)={\sum }_q{\kappa }_{X_i|Q}\left(x_i\right|q)P_{Q|Y}\left(q\right|y)$ (such a channel must exist since $X_i{\prec }_{Y}Q$). Define the random variables ${\tilde{X}}_1,\dots ,{\tilde{X}}_n$ with the joint distribution

$$P_{YQ{\tilde{X}}_1\dots {\tilde{X}}_n}(y,q,x_1,\dots ,x_n)=P_Y\left(y\right)P_{Q|Y}\left(q\right|y)\prod _i{\kappa }_{X_i|Q}\left(x_i\right|q).$$ (A24)

Note that the pairwise marginals obey $P_{{\tilde{X}}_{i}Y}=P_{X_{i}Y}$. Thus, all of the ${\tilde{X}}_i$ satisfy the marginal constraints in the right hand side of Equation (31), so

$$I_{\cup }^{\mathrm{BROJA}}(X_1;\dots ;X_n\to Y)\le I(Y;{\tilde{X}}_1,\dots ,{\tilde{X}}_n).$$ (A25)

By elementary properties of mutual information, we have

$$I(Y;{\tilde{X}}_1,\dots ,{\tilde{X}}_n)\le I(Y;Q,{\tilde{X}}_1,\dots ,{\tilde{X}}_n)$$ (A26)

Given Equation (A24), the Markov condition $Y-Q-{\tilde{X}}_1,\dots ,{\tilde{X}}_n$ holds, so

$$I(Y;{\tilde{X}}_1,\dots ,{\tilde{X}}_n)\le I(Y;Q)$$ (A27)

by the data processing inequality. Combining Equations (A23) and (A25) to (A27) implies

$$I_{\cup }^{\mathrm{BROJA}}(X_1;\dots ;X_n\to Y)\le I(Y;Q)\le I_{\cup }^{\prec }(X_1;\dots ;X_n\to Y)+ϵ.$$

Since this holds for all $ϵ$, we can take the limit $ϵ\to 0$ to give $I_{\cup }^{\mathrm{BROJA}}(X_1;\dots ;X_n\to Y)\le I_{\cup }^{\prec }(X_1;\dots ;X_n\to Y)$. The result follows by combining with Equation (A22). □

Appendix H. Relation between $I_{\cap }^{\mathrm{WB}}$ and Our General Framework

Here we consider whether the redundancy measure $I_{\cap }^{\mathrm{WB}}$ proposed by Williams and Beer [11] can be put in the general form Equation (7). This measure is defined as

$$I_{\cap }^{\mathrm{WB}}(X_1;\dots ;X_n\to Y):=\sum _y{P}_Y\left(y\right)\underset{i}{\min }I(X_i;Y=y),$$ (A28)

where $I(X_i;Y=y)$ is called the “specific information” between $X_i$ and target outcome y,

$$I(X_i;Y=y):=D_{\mathrm{KL}}(P_{X_i|Y=y}\parallel P_{X_i})=\sum _{x_i}{P}_{X_i|Y}\left(x_i\right|y)log\frac{P_{X_i|Y}\left(x_i\right|y)}{P_{X_i}\left(x_i\right)},$$

and $D_{\mathrm{KL}}$ is Kullback-Leibler (KL) divergence.

Specific information obeys $I(X;Y)={\sum }_y{P}_Y\left(y\right)I(X;Y=y)$. Thus, Equation (A28) looks similar to a mutual information expression, where each specific information term is given by the smallest specific information that y carries about any of the sources. Motivated by this interpretation, one might ask whether there exists a random variable Q whose specific information terms are equal to $I(Q;Y=y)={\min }_{i}I(X_i;Y=y)$ for each y. If such a random variable existed, then $I_{\cap }^{\mathrm{WB}}$ could be written as

$$I_{\cap }^{\mathrm{WB}}(X_1;\dots ;X_n\to Y)\stackrel{?}{=}\underset{Q}{\max }I(Q;Y)\mathrm{such}\mathrm{that}\forall i,y:I(Q;Y=y)\le I(X_i;Y=y),$$ (A29)

which has the form of Equation (7), with the ⊏ order defined as

$$A⊏B\mathrm{iff}I(A;Y=y)\le I(B;Y=y)\mathrm{for}\mathrm{all}y\in \mathcal{Y}.$$ (A30)

Here we provide a counterexample to demonstrate that such a variable does not exist in general, and so therefore Equation (A29) is not generally valid. Suppose Y has three outcomes $\mathcal{Y}=\{0,1,2\}$ with a uniform distribution, and consider two binary sources $X_1,X_2$ with the following conditional distributions,

$$\begin{array}{cc}\hfill P_{X_1|Y}\left(x_1\right|y)& =\left\{\begin{array}{cc}\delta (x_1,y)\hfill & \mathrm{if}y\in \{0,1\}\hfill \\ \frac{1}{2}\delta (x_1,0)+\frac{1}{2}\delta (x_1,1)\hfill & \mathrm{if}y=2\hfill \end{array}\right.\hfill \\ \hfill P_{X_2|Y}\left(x_1\right|y)& =\left\{\begin{array}{cc}\delta (x_1,y)\hfill & \mathrm{if}y\in \{0,2\}\hfill \\ \frac{1}{2}\delta (x_1,0)+\frac{1}{2}\delta (x_1,2)\hfill & \mathrm{if}y=1\hfill \end{array}\right.\hfill \end{array}$$

In this case, a simple calculation shows that the specific information obeys (in bits)

$$\begin{array}{cc}\hfill I(X_1;Y=0)=1& I(X_2;Y=0)=1\hfill \\ \hfill I(X_1;Y=1)=1& I(X_2;Y=1)=0\hfill \\ \hfill I(X_1;Y=2)=0& I(X_2;Y=2)=1\hfill \end{array}$$

Plugging into Equation (A28) gives $I_{\cap }^{\mathrm{WB}}(X_1;X_2\to Y)=1/3$.

Now consider the optimization problem in Equation (A29). Since $I(X_1;Y=2)=I(X_2;Y=1)=0$, any allowed Q must satisfy $I(Q;Y=1)=I(Q;Y=2)=0$ and therefore $P_{Q|Y=1}=P_Q=P_{Q|Y=2}$. Combined with the marginalization identity $P_Y\left(0\right)P_{Q|Y=0}+P_Y\left(1\right)P_{Q|Y=1}+P_Y\left(2\right)P_{Q|Y=2}=P_Q$, this implies that $P_{Q|Y=0}=P_Q$ and therefore that $I(Q;Y=0)=0$. Thus, any allowed Q obeys $I(Q;Y)=0\ne I_{\cap }^{\mathrm{WB}}$. This means that $I_{\cap }^{\mathrm{WB}}$ cannot be expressed in the form of Equation (7) when ⊏ is defined as Equation (A30).

Appendix I. Miscellaneous Derivations

Proof of Lemma 1. 

We use a modified version of the example in [39,68]. Consider a set of $n\ge 3$ sources. The inclusion-exclusion principle states that

$$I_{\cup }(X_1;\dots ;X_n\to Y)=\sum _{J\subseteq \{1,\dots ,n\}\setminus \{⌀\}}{(-1)}^{\left|J\right|-1}{I}_{\cap }(X_{J_1};\dots ;X_{J_{\left|J\right|}}\to Y).$$ (A31)

Now, let $X_1,\dots ,X_{n-1}$ be uniformly distributed and statistically independent binary random variables, and take $X_n=X_1\mathrm{XOR}{X}_2$ and $Y=(X_1,X_2,X_n)$. Note that $I(Y;X_i)=1\mathrm{bit}$ for $i\in \{1,2,n\}$ and $I(Y;X_i)=0$ for $i\in \{3,\dots ,n-1\}$, and that $I(Y;X_1,\dots ,X_n)=2\mathrm{bit}$. Thus, $I_{\cap }(X_i;X_j\to Y)=0$ whenever $i\in \{3,\dots ,n-1\}$ or $j\in \{3,\dots ,n-1\}$, as follows from Symmetry, Self-redundancy, and Monotonicity. Note also that the outcomes of Y are simply a relabelling of $(X_1,X_2)$, and similarly for $(X_1,X_n)$ and $(X_2,X_n)$. Then, since by Independent identity property, $I_{\cap }(X_i;X_j\to Y)=0$ for $i\ne j$ where $i,j\in \{1,2,n\}$. Thus, $I_{\cap }(X_i;X_j\to Y)=0$ for all pairs $i\ne j$. By Monotonicity, redundancy is 0 for any set of 2 or more sources.

Plugging this into Equation (A31) gives

$$I_{\cup }(X_1;\dots ;X_n\to Y)=\sum _i{I}_{\cap }(X_i\to Y)=\sum _{i}I(X_i\to Y)=3\mathrm{bit}$$

Note that this exceeds the total amount of information about the target provided jointly by all sources, which is only 2 bits, so $I_{\cup }(X_1;\dots ;X_n\to Y)\neg \le I(Y;X_1,\dots ,X_n)$. □

Proof of Theorem 3. 

Without loss of generality, let $i=1$. We will use that $U^{\prec }(X_1\to Y|X_1;\dots ;X_n)=0$ is equivalent to

$$I_{\cap }^{\prec }(X_1;\dots ;X_n\to Y)=I(X_1;Y).$$ (A32)

We will use that by monotonicity of mutual information with respect to ${\prec }_Y$ (see Section 4.1),

$$I(X_1;Y)\ge I_{\cap }^{\prec }(X_1;\dots ;X_n\to Y).$$ (A33)

We first prove the “if” direction. Since $Q=X_1$ is in the feasible set of Equation (27), $I_{\cap }^{\prec }(X_1;\dots ;X_n\to Y)\ge I(X_1;Y)$. Combining with Equation (A33) gives Equation (A32).

We now prove the “only if” direction. As described in Appendix C, $I_{\cap }^{\prec }$ can be expressed as an optimization over a finite dimensional, closed, and bounded region, so the supremum in Equation (27) is achieved. Thus, there is some Q such that $Q{\prec }_Y{X}_i$ for all i and

$$\begin{array}{cc}\hfill I(Y;Q)& =I_{\cap }^{\prec }(X_1;\dots ;X_n\to Y).\hfill \end{array}$$ (A34)

Since $Q{\prec }_Y{X}_1$, there is a conditional probability distribution ${\kappa }_{\mathcal{Q}|{\mathcal{X}}_1}$ such that $P_{Q|Y}\left(q\right|y)={\sum }_{x_1}{\kappa }_{Q|X_1}\left(q\right|x_1)P_{X_1|Y}\left(x_1\right|y)$. Define a random variable $\tilde{Q}$ with the joint distribution

$$P_{Y{X}_1\tilde{Q}}(y,x_1,q)={\kappa }_{Q|X_1}\left(q\right|x_1)P_{Y{X}_1}(y,x_1).$$

We will use that $P_{QY}=P_{\tilde{Q}Y}$. Then, the chain rule for mutual information gives

$$\begin{array}{cc}\hfill I(Y;X_1,\tilde{Q})& =I(Y;\tilde{Q})+I(Y;X_1|\tilde{Q})=I(Y;X_1)+I(Y;\tilde{Q}|X_1)=I(Y;X_1),\hfill \end{array}$$

where we used the Markov condition $Y-X_1-\tilde{Q}$. Combining and rearranging gives

$$I(Y;\tilde{Q})=I(Y;X_1)-I(Y;X_1|\tilde{Q}).$$ (A35)

Now assume that Equation (A32) holds. Combining with Equation (A34) and $P_{QY}=P_{\tilde{Q}Y}$ gives $I(Y;X_1)=I(Y;Q)=I(Y;\tilde{Q})$. Combining with Equation (A35) gives $I(Y;X_1|\tilde{Q})=0$, meaning that the Markov condition $Y-\tilde{Q}-X_1$ holds and therefore $X_1{\prec }_Y\tilde{Q}$. Since $Q{\prec }_Y{X}_i$ for all i and $P_{QY}=P_{\tilde{Q}Y}$, it also the case that $\tilde{Q}{\prec }_Y{X}_i$ for all i. Finally, since ${\prec }_Y$ is transitive, $X_1{\prec }_Y{X}_i$ for all i, which is the desired result. □

Proof of Theorem 4. 

Without loss of generality, let $i=1$. We will use that $E^{\prec }(X_1\to Y|X_1;\dots ;X_n)=0$ is equivalent to

$$I_{\cup }^{\prec }(X_1;\dots ;X_n\to Y)=I(X_1;Y).$$ (A36)

We will use that by monotonicity of mutual information with respect to ${\prec }_Y$ (see Section 4.1),

$$I(X_1;Y)\le I_{\cup }^{\prec }(X_1;\dots ;X_n\to Y).$$ (A37)

We first prove the “if” direction. Since $Q=X_1$ is in the feasible set of Equation (28), $I_{\cup }^{\prec }(X_1;\dots ;X_n\to Y)\le I(X_1;Y)$. Combining with Equation (A37) gives Equation (A36).

We now prove the “only if” direction. As we show in Appendix G, $I_{\cup }^{\prec }$ is equivalent to $I_{\cup }^{\mathrm{BROJA}}$, which is defined as an optimization over a finite dimensional, closed, and bounded region. Thus the infimum in Equation (28) is always achieved, so there is some Q such that $X_i{\prec }_{Y}Q$ for all i and

$$\begin{array}{cc}\hfill I(Y;Q)& =I_{\cup }^{\prec }(X_1;\dots ;X_n\to Y).\hfill \end{array}$$ (A38)

Moreover, since $X_1{\prec }_{Y}Q$, there is a conditional probability distribution ${\kappa }_{X_1|Q}$ such that $P_{X_1|Y}\left(x_1\right|y)={\sum }_q{\kappa }_{X_1|Q}\left(x_1\right|q)P_{Q|Y}\left(q\right|y)$. Define a random variable ${\tilde{X}}_1$ with the joint distribution

$$P_{Y{\tilde{X}}_{1}Q}(y,x_1,q)={\kappa }_{X_1|Q}\left(x_1\right|q)P_{QY}(q,y).$$

We will use that $P_{X_{1}Y}=P_{{\tilde{X}}_{1}Y}$. Then, using the chain rule for mutual information,

$$\begin{array}{cc}\hfill I(Y;{\tilde{X}}_1,Q)& =I(Y;{\tilde{X}}_1)+I(Y;Q|{\tilde{X}}_1)=I(Y;Q)+I(Y;{\tilde{X}}_1|Q)=I(Y;Q),\hfill \end{array}$$

where we used the Markov condition $Y-Q-{\tilde{X}}_1$. Combining and rearranging gives

$$I(Y;{\tilde{X}}_1)=I(Y;Q)-I(Y;Q|{\tilde{X}}_1).$$ (A39)

Now assume that Equation (A36) holds. Combining with Equation (A38) and $P_{X_{1}Y}=P_{{\tilde{X}}_{1}Y}$ gives $I(Y;X_1)=I(Y;{\tilde{X}}_1)=I(Y;Q)$. Combining with Equation (A39) gives $I(Y;Q|{\tilde{X}}_1)=0$, meaning that the Markov condition $Y-{\tilde{X}}_1-Q$ holds and therefore $Q{\prec }_Y{\tilde{X}}_1$. Since $P_{X_{1}Y}=P_{{\tilde{X}}_{1}Y}$, it is also the case that $Q{\prec }_Y{X}_1$. Finally, since $X_i{\prec }_{Y}Q$ for all i and ${\prec }_Y$ is transitive, $X_i{\prec }_Y{X}_1$ for all i, which is the desired result. □

Proof of Theorem 6. 

Consider any random variable Q which achieves the maximum in Equation (27). This implies there are channels ${\kappa }_{Q|X_1}$ and ${\kappa }_{Q|X_2}$ such that for any $q\in \mathcal{Q}$ and $(x_1,x_2)\in {\mathcal{X}}_1\times {\mathcal{X}}_2$ with $p_{X_1{X}_2}(x_1,x_2)>0$,

$$\begin{array}{cc}\hfill P_{Q|X_1{X}_2}\left(q\right|x_1,x_2)& =\sum _{x_1^{\prime }}{\kappa }_{Q|X_1}\left(q\right|x_1^{\prime })P_{X_1|X_1{X}_2}\left(x_1^{\prime }\right|x_1,x_2)\hfill \\ \hfill P_{Q|X_1{X}_2}\left(q\right|x_1,x_2)& =\sum _{x_2^{\prime }}{\kappa }_{Q|X_2}\left(q\right|x_2^{\prime })P_{X_2|X_1{X}_2}\left(x_2^{\prime }\right|x_1,x_2).\hfill \end{array}$$

We now equate the above two expressions, while using that $P_{X_1|X_1{X}_2}\left(x_1^{\prime }\right|x_1,x_2)=\delta (x_1^{\prime },x_1)$ and $P_{X_2|X_1{X}_2}\left(x_2^{\prime }\right|x_1,x_2)=\delta (x_2^{\prime },x_2)$ (where $\delta (·,·)$ is the Kronecker delta). This gives

$${\kappa }_{Q|X_1}\left(q\right|x_1)={\kappa }_{Q|X_2}\left(q\right|x_2)$$ (A40)

for all q and any $(x_1,x_2)$ where $p_{X_1{X}_2}(x_1,x_2)>0$.

Now consider a bipartite graph with vertex set ${\mathcal{X}}_1\cup {\mathcal{X}}_2$ and an edge between vertex $x_1$ and vertex $x_2$ if $P_{X_1{X}_2}(x_1,x_2)>0$. Define $\mathsf{\Pi }$ to be the set of connected components of this bipartite graph, and let $f_1:{\mathcal{X}}_1\to \mathsf{\Pi }$ be a function that maps each $x_1$ to its corresponding connected component (for any $x_1$ with $P_{X_1}\left(x_1\right)=0$, $f_1\left(x_1\right)$ can be any value). Equation (A40) implies that if $x_1$ and $x_1^{\prime }$ both belong to the same connected component, then the constraint Equation (A40) will “propagate” from $x_1$ to $x_1^{\prime }$, so that ${\kappa }_{Q|X_1}\left(q\right|x_1)={\kappa }_{Q|X_1}\left(q\right|x_1^{\prime })$. Said differently, this means that ${\kappa }_{Q|X_1}\left(q\right|x_1)={\kappa }_{Q|X_1}\left(q\right|f_1\left(x_1\right))$ and that the Markov condition $(X_1,X_2)-X_1-f_1\left(X_1\right)-Q$ holds. This gives

$$I(X_1,X_2;Q)\le I(X_1,X_2;f_1\left(X_1\right))=H\left(f_1\left(X_1\right)\right),$$ (A41)

where the first inequality uses the data processing inequality, and the second equality uses that $f_1\left(X_1\right)$ is a deterministic function of $(X_1,X_2)$. The upper bound in Equation (A41) is achieved when $Q=f_1\left(X_1\right)$, thus $Q⊲X_1$. A similar argument shows that $Q⊲X_2$.

We have shown that the constraints in Equation (27) can be replaced by $Q⊲X_1,Q⊲X_2$. It is also clear that any Q which is a deterministic function of either $X_1$ or $X_2$ must also be a deterministic function of the target $Y=(X_1,X_2)$, hence $I(Y;Q)=H\left(Q\right)$. Combining these results shows that Equation (27) is equivalent to Equation (32) for the COPY gate. □

Funding

This research received no external funding.

Data Availability Statement

Not applicable.

Conflicts of Interest

The authors declare no conflict of interest.