Communication Efficient Algorithms for Bounding and Approximating the Empirical Entropy in Distributed Systems

Abstract

The empirical entropy is a key statistical measure of data frequency vectors, enabling one to estimate how diverse the data are. From the computational point of view, it is important to quickly compute, approximate, or bound the entropy. In a distributed system, the representative (“global”) frequency vector is the average of the “local” frequency vectors, each residing in a distinct node. Typically, the trivial solution of aggregating the local vectors and computing their average incurs a huge communication overhead. Hence, the challenge is to approximate, or bound, the entropy of the global vector, while reducing communication overhead. In this paper, we develop algorithms which achieve this goal.

1. Introduction

Consider the distributed computing model [1,2,3], where the goal is to compute a function over input divided amongst multiple nodes. A local computation, while simple, does not always suffice to reach a conclusion on the aggregated input data, especially when the function is nonlinear. On the other hand, broadcasting the local data to a coordinator node is impractical and undesirable due to communication overhead, energy consumption, and privacy issues. Generally, we seek to approximate or bound the function’s value on the aggregated data without broadcasting it in its entirety.

The function we handle in this paper is the empirical Shannon entropy [4], which is defined as ${\sum }_i-x_{i}ln\left(x_i\right)$ for a frequency vector $X=\left(x_1,\dots ,x_n\right)$ (i.e., all values are non-negative and sum to 1). For some of the ensuing analysis, it is easier to use the natural logarithm than the base 2 one, which only changes the value by a multiplicative constant. Thus, hereafter, “entropy” will refer exclusively to the empirical Shannon entropy. Specifically, we assume there exists a distributed system, with each node (or “party”) holding a “local” frequency vector. The target function is defined as the global system entropy, which is equal to the empirical Shannon entropy of the average of the local vectors. Alas, to compute the exact value, we must first aggregate the local vectors and average them, which often incurs a huge communication overhead. Fortunately, it often suffices to approximate, or bound, this global entropy; for example:

• Often, a sudden change in the entropy indicates a phase change in the underlying system, for example a DDoS (Distributed Denial of Service) attack [5]. To this end, it typically suffices to bound the entropy, since its precise value is not required.

• A good measure of similarity between two datasets is the difference between their aggregated and individual entropies. For example, if two collections of text are of a similar nature, their aggregate entropy will be similar to the individual ones, and if they are different, the aggregated entropy will be substantially larger. Here, too, it suffices to approximate the global entropy, or bound it from above or below in order to reach a decision.

Guided by such challenges, we develop communication-efficient algorithms for bounding and approximating the global entropy, which are organized as follows:

In Section 3, we present algorithms for bounding the global entropy, with low communication. Some results on real and synthetic data are also provided.

In Section 4, a novel algorithm is provided for approximating the global entropy. It is tailored to treat the cases in which the algorithm in Section 3 underperforms.

2. Previous Work

The problem of reducing communication overhead in distributed systems is very important both from the practical and theoretical points of view. Applications abound, for example distributed graphs [2,6] and distributed machine learning [3]. Research close to ours in spirit [7] deals with the following scenario: a system is given consisting of

• Distributed computing nodes denoted by $N_1\dots N_t$, with $N_i$ holding a “local” data vector $X_i$. The nodes can communicate, either directly or via a “coordinator” node.

• A scalar-valued function $f(X,Y)$, defined on pairs of real-valued vectors.

Given the above, the challenge is to approximate the values $f(X_i,X_j),i,j=1\dots t$ with a low communication overhead; that is, the trivial solution of sending all the local vectors to some computing node is forbidden.

A sketch for this type of problem is defined as a structure $s\left(\right)$, of size smaller than the dimension of $X_i$, which has the following property: knowledge of $s\left(X\right),s\left(Y\right)$ allows one to approximate $f(X,Y)$ with very high accuracy. An important example [7] is $f(X,Y)=\langle X,Y\rangle$ ($\langle X,Y\rangle$ stands for the inner product of $X,Y$).

There are many types of sketches, for example:

• PCA (Principle Component Analysis) sketch: given a large subset $S\subseteq {\mathbb{R}}^n$, one wishes to quickly estimate the distance of vectors from an underlying structure which S is sampled from (a famous example is images of a certain type [8]). To this end, S is represented by a smaller set, consisting of the dominant eigenvectors of S’s scatter matrix, and the distance is estimated by the distance of the vector from the subspace spanned by these vectors.

• In the analysis of streaming data, some important sketches were developed, in order to handle large and dynamic streams, by only preserving salient properties (such as the number of distinct items, frequency, and the norm). It is beyond the scope of this paper to describe these sketches, so we refer to the literature [9].

Sketches are specifically tailored for the task at hand. In our case, $X,Y$ are frequency (probability) vectors, and $f(X,Y)$ is the empirical Shannon entropy of $\left(X+Y\right)/2$. Similarly, one may look at functions defined on larger subsets of $\{X_1,\dots ,X_t\}$ (Section 3.4). Our task is therefore to define a sketch $s\left(\right)$, such that

• $s\left(X\right)$ is much smaller than X.

• Knowledge of $s\left(X\right),s\left(Y\right)$ allows one to approximate the empirical Shannon entropy of $\left(X+Y\right)/2$.

We note here that some work addressed entropy approximation in the Streaming Model  [1,10,11]. Here, as in [7], we are mainly interested in the static scenario, in which the overall communication overhead is substantially smaller than the overall data volume. The “geometric monitoring” method [10,11], applied to solve the Distributed Monitoring Problem [1], relies on checking local constraints at the nodes; as long as they hold, the value of some global function, defined on the average of the local streams, is guaranteed to lie in some range. Alas, when the local conditions are violated, the nodes undertake a “synchronization stage” [12], which consists of communicating their local vectors in their entirety (which here we avoid). In the future, we plan to extend the techniques developed here to the distributed streaming scenario.

3. Dynamic Bounds and Communication Reduction

In this section, we present algorithms for bounding the entropy of a centralized vector—that is, the mean of several local vectors—by broadcasting a controlled amount of inter-communication between machines. The proposed algorithms for both upper and lower bounds accept the same input and therefore can be run concurrently.

3.1. Problem, Motivation, and an Example

This work addresses the following problem:

• Given nodes $N_i$, each holding a probability vector $v_i$ (i.e., all values are positive and sum to 1), approximate the entropy of the average of $v_i$, while maintaining low communication overhead.

Let us start with the simplest possible scenario, which we shall analyze in detail, in order to prepare the ground for the general treatment.

Example 1. 

There are two nodes, $N_1,N_2$, and the vectors they hold are of length 3. Assume without loss of generality that $N_1$ sends some of its data to $N_2$, where “data” consists of a set of pairs (coordinate, value), where “coordinate” is the location of a value of $v_1$, and “value” is its numerical value; then, $N_2$ attempts to derive an upper bound on the entropy of $\frac{v_1+v_2}{2}$. Note that vectors of length 2 are hardly interesting, since sending a single datum allows one to compute the other (as they sum to 1); hence, $N_2$ will be able to exactly compute the entropy.

Intuition suggests that $N_1$ should relay its largest value to $N_2$. While (as we will show later), this is true on the average, that is not always the case. Assume that the vectors held by the nodes are

$$v_1=\left(\frac{2}{3},\frac{1}{3},0\right),v_2=\left(0,\frac{1}{2},\frac{1}{2}\right)$$

Assume that $N_1$ sends its largest value (and its coordinate) to $N_2$. Now, $N_2$ knows that (a) the first value of the average vector is $\frac{1}{3}$, and (b) the second and third values of $N_2$ sum to $\frac{1}{3}$. That leaves open the possibility that these values are $\frac{1}{6}$ each, which would render the average vector equal to

$$\left(\frac{1}{3},\frac{1}{3},\frac{1}{3}\right),$$

hence, the upper bound is equal to the maximal entropy possible, $3ln\left(3\right)$. However, if $N_1$ sends its second largest value $\left(\frac{1}{3}\right)$ to $N_2$, $N_2$ can conclude that the second value of the average vector equals $\frac{5}{12}$; hence, the upper bound on the entropy is strictly smaller than $3ln\left(3\right)$.

We observe here that the key consideration in determining the upper bound is the distribution of the “slack” corresponding to the unknown values at the other node ($N_1$ in this example). The overall size of this “slack” is one-half of the unknown values, and it should be distributed amongst the same set of coordinates in $N_2$ after they have been divided by 2.

In contrast to the above “adversarial” example, on average, it is optimal to send the largest value (i.e., it allows one to achieve a lower upper bound). To prove this, we have (numerically) computed the integral of the upper bound over all triplets, both after the largest value and a random value were sent; sending the highest value, on the average, yielded an upper bound lower by 0.041 than sending a random value. More general experiments, for both real and synthetic data, are reported in Section 3.5.

We now address the general scenario. Let us start with a few definitions:

Notation 1. 

Let $\{X_1,\dots ,X_t\}$ be a set of local vectors held in t nodes $\{N_1,\dots ,N_t\}$. Then, $\tilde{X}=\frac{1}{t}{\sum }_{i=1}^t{X}_i$ is the aggregate vector, which in our case is the mean over all local vectors.

Notation 2. 

Let $x\in [0,1]$. We define the Entropy activation function $h\left(x\right)$ by:

$$h\left(x\right)=\left\{\begin{array}{cc}0\hfill & \mathit{if}x=0\hfill \\ -xlnx\hfill & \mathit{otherwise}\hfill \end{array}\right.$$

Definition 1. 

Let $X\in {\mathbb{R}}^n$ s.t $\forall i:0\le x_i\le 1$. Then, $H\left(X\right)$ denotes the Shannon’s Entropy of X [4], given by

$$H\left(X\right)=\sum _{i=1}^{n}h\left(x_i\right)$$

We will henceforth assume all vectors are of length n and behave like X in Definition 1, even if it is not explicitly noted. We also assume each value of X can be represented by at most b bits.

Notation 3. 

Let $X_{\mathit{Local}},X_{\mathit{Other}}$ denote a probability vector held by a local machine and a probability vector held by a remote machine, respectively.

In this section, we present algorithms for deciding whether the entropy of an average probability vector that sums to 1 is greater or lesser than a user-defined threshold. Formally, we will address two problems:

1. Determining whether the inequality $H\left(\tilde{X}\right)\le L$ holds for some user-defined constant L.

2. Determining whether the inequality $H\left(\tilde{X}\right)\ge U$ holds for some user-defined constant U.

We begin with a lemma which provides the foundation for both the Local Upper Bound (Section 3.2) and Local Lower Bound (Section 3.3) in the following subsections.

While noting that the lemma and its corollary hold for any vector $X\in {\mathbb{R}}^n$, our vectors are always frequency vectors and hence sum to 1; the $\mathsf{\Delta }$ below corresponds to the “slack” added after dividing the respective value by 2, as explained in the discussion of Example 1 above; hence, the values still sum to 1.

Lemma 1 

(Extrema of Entropy). Let $X=\left(x_1,\dots ,x_n\right)\in {\mathbb{R}}^n$ s.t. $\forall i,x_i\ge 0$. Let Δ be a positive number, and let $i,j$ be two distinct coordinates of X.

• Let $X^i=\left(x_1,\dots ,x_i+\mathsf{\Delta },\dots ,x_n\right)$;

• Let $X^j=\left(x_1,\dots ,x_j+\mathsf{\Delta },\dots ,x_n\right)$.

If $x_i<x_j$, then $H\left(X^i\right)>H\left(X^j\right)$.

Proof. 

To establish $H\left(X^i\right)-H\left(X^j\right)>0$, then since $H\left(X\right)$ is coordinate-wise additive, it suffices to show that:

$$h(x_i+\mathsf{\Delta })-h\left(x_i\right)>h(x_j+\mathsf{\Delta })-h\left(x_j\right).$$

Using the observation that $h^{\prime }\left(x\right)=-lnx-1$, which is strictly decreasing, we divide the proof into two cases depending on the relation between $x_i+\mathsf{\Delta }$ and $x_j$:

• 1.$x_i+\mathsf{\Delta }\le x_j$:
  Since $x_i<x_i+\mathsf{\Delta }\le x_j<x_j+\mathsf{\Delta }$, the intervals $(x_i,x_i+\mathsf{\Delta })$ and $(x_j,x_j+\mathsf{\Delta })$ are disjoint. By applying the Lagrange Mean Value Theorem, for some $c_1\in (x_i,x_i+\mathsf{\Delta })$ and $c_2\in (x_j,x_j+\mathsf{\Delta })$:

  $$\begin{array}{cc}\hfill h^{\prime }\left(c_1\right)& =\frac{h(x_i+\mathsf{\Delta })-h\left(x_i\right)}{\mathsf{\Delta }}\hfill \\ \hfill h^{\prime }\left(c_2\right)& =\frac{h(x_j+\mathsf{\Delta })-h\left(x_j\right)}{\mathsf{\Delta }}\hfill \end{array}$$
  Since $h^{\prime }\left(\right)$ is decreasing and $c_1<c_2$, we immediately obtain that $h^{\prime }\left(c_1\right)>h^{\prime }\left(c_2\right)$. It follows that:

  $$\begin{array}{cc}\hfill \frac{h(x_i+\mathsf{\Delta })-h\left(x_i\right)}{\mathsf{\Delta }}=h^{\prime }\left(c_1\right)& >h^{\prime }\left(c_2\right)=\frac{h(x_j+\mathsf{\Delta })-h\left(x_j\right)}{\mathsf{\Delta }}\hfill \\ \hfill h(x_i+\mathsf{\Delta })-h\left(x_i\right)& >h(x_j+\mathsf{\Delta })-h\left(x_j\right)\hfill \end{array}$$
• 2.$x_i+\mathsf{\Delta }>x_j$:
  Observing the disjoint intervals $(x_i,x_j)$, $(x_i+\mathsf{\Delta },x_j+\mathsf{\Delta })$. The sought inequality, following the Lagrange Mean Value Theorem for $c_1\in (x_i,x_j),c_2\in (x_i+\mathsf{\Delta },x_j+\mathsf{\Delta })$, as in the case above, is:

  $$\begin{array}{cc}\hfill \frac{h\left(x_j\right)-h\left(x_i\right)}{\mathsf{\Delta }}=h^{\prime }\left(c_1\right)& >h^{\prime }\left(c_2\right)=\frac{h(x_j+\mathsf{\Delta })-h(x_i+\mathsf{\Delta })}{\mathsf{\Delta }}\hfill \\ \hfill h(x_i+\mathsf{\Delta })-h\left(x_i\right)& >h(x_j+\mathsf{\Delta })-h\left(x_j\right)\hfill \end{array}$$

   □

Corollary 1. 

Given a probability vector X, and $\mathsf{\Delta }>0$, the following properties hold:

• 1.  If Δ is added to any value of X, the maximal increase of its entropy will occur when Δ is added to the minimal value of X.
• 2.  If Δ is added to any value of X, the minimal increase of its entropy will occur when Δ is added to the maximal value of X.

3.2. Upper Bound

While $ln\left(n\right)$ is a trivial upper bound to the entropy, and does not require any communication to agree upon, we can develop a more efficient alternative while incurring a small communication overhead. Let $X,S_k\left(X\right)$ denote a probability vector and a k-sized ordered subset of X’s k largest values, respectively. Hence, let local nodes broadcast the following two ordered sets:

1. $S_k\left(X\right)=$ ordered set of largest k values of X;

2. $C_k\left(X\right)=$ the coordinates of the values in $S_k\left(X\right)$, or formally $\{i\mid x_i\in S_k\left(X\right)\}$.

Each of these messages costs at most $k(b+{log}_{2}n)$ bits: b for each value and ${log}_{2}n$ for each corresponding coordinate. By sending these subsets of values and coordinates, local machines can immediately obtain the following information regarding the local vector X from which $S_k\left(X\right),C_k\left(X\right)$ were sent:

• The sum of all values not in $S_k\left(X\right)$, i.e., $1-{\sum }_{x\in S_k\left(X\right)}x$, will be referred to as the mass of the local vector that remains available to be distributed among coordinates. It will be denoted by m in the following algorithms.

• $max\{X\backslash S_k\left(X\right)\}\le min\left\{S_k\left(X\right)\right\}$, since $S_k\left(X\right)$ contains the largest values of X (where $X\backslash S_k\left(X\right)$ denotes set difference).

We next suggest an algorithm for a local machine with local probability vector $X_{\mathrm{Local}}$ to compute the strict upper bound for $\tilde{X}$, which is the aggregated data of both $X_{\mathrm{Local}}$ and $X_{\mathrm{Other}}$, which is a probability vector that is not accessible to the machine. The remote machine broadcasts $S_k\left(X_{\mathrm{Other}}\right)$ and $C_k\left(X_{\mathrm{Other}}\right)$ for some predetermined k.

The algorithm constructs the unknown subset of the remote machine that ensures the centralized entropy is maximized, or formally ${argmax}_{X}H\left(X_{\mathrm{Local}}+X\right)$, while maintaining feasible constraints. We view this problem as an instance of constrained optimization, where our target function is the global entropy, and the constraints are given by the broadcast set of $S_K\left(X\right)$ and its sum. The main tool is Corollary 1 for every coordinate of $X_{\mathrm{Local}}$.

Before we present the algorithm, we note two extreme cases, which instantly produce an upper bound without the need to algorithmically compute it:

• 1.${\sum }_{x\in S_k\left(X\right)}x\approx 1$. In this case, most (or all) the information of X is broadcast by the message, and the entropy can be computed accurately without need for a bound.
• 2.$1-{\sum }_{x\in S_k\left(X\right)}x\ge {\sum }_{x_i\in X_{\mathrm{Local}}}{x}_{max}-x_i$, where $x_{max}=max\left\{X_{\mathrm{Local}}\right\}$. In this case, there is no need to run the proposed algorithm; the constraint maximization will always result is an “optimal” target—the uniform vector with the value $x_{max}+\frac{1}{n}\left(m-{\sum }_{x_i\in X_{\mathrm{Local}}}{x}_{max}-x_i\right)$, whose entropy we know is maximal w.r.t its sum.

Theorem 1. 

Algorithm 1 runs in $O\left(n^2\right)$ time and returns an upper bound on the entropy of $\tilde{X}=\frac{1}{2}\left(X_{\mathit{Local}}+X_{\mathit{Other}}\right)$.

Algorithm 1: Upper Entropy Bound for Two Nodes

Proof. 

Let $n^{\prime }$ be the length of $X^{*}$, which equals $n-k$. In each loop iteration, the algorithm increments no more than $n^{\prime }$ values of $X^{*}$, and since there are $n^{\prime }$ coordinates of $X^{*}$, it will perform at most $n^{\prime }$ steps. Hence, the bound of $O\left(n^2\right)$ runtime follows.

Let $X^{*}=(x_1,\dots ,x_{n^{\prime }})$ be the initial vector as noted in line 3, and let Y denote the same by the end of the while loop, i.e., after the condition $m=0$ is met. Let the coordinates of Y be arranged in ascending order, which has no effect on its entropy: $Y=\left(y_1,y_2,\dots ,y_t,y_{t+1},\dots ,y_{n^{\prime }}\right)$. Since at every loop iteration, all minimal coordinates are incremented simultaneously, there exists some coordinate t such that for all $i<t$, $y_i$ equals c, and for all $i>t$, $y_i$ is strictly greater than c. Hence, we can view Y as a concatenation of the two vectors $(Y_L,Y_R)$ as defined below:

• $Y_L=\left(y_1,\dots ,y_t\right)=\left(c,\dots ,c\right)$;

• $Y_R=\left(y_{t+1},\dots ,y_{n^{\prime }}\right)$.

Let $s\left(X\right)$ denote the sum of X. It now suffices to show that any vector Z that sums to $s\left(X^{*}\right)+m$ and can be achieved by performing only additions to $X^{*}$ has a lesser or equal entropy value than Y. Let Z denote such a vector for every value of which $z_i$ satisfies $z_i\ge x_i$. Let $Z=(Z_L,Z_R)$, where $Z_L=(z_1,\dots ,z_t),Z_R=(z_{t+1},\dots ,z_{n^{\prime }})$ for the same t as defined above.

Since it holds that $s\left(Z\right)=s\left(Z_L\right)+s\left(Z_R\right)=s\left(Y_L\right)+s\left(Y_R\right)=s\left(Y\right)$, we examine the following cases:

• $s\left(Z_L\right)=s\left(Y_L\right),s\left(Z_R\right)=s\left(Y_R\right)$: Note that $Z_R=Y_R$, since their sum is equal, and $Y_R$ has had no further additions. In addition, since $s\left(Z_L\right)=s\left(Y_L\right)=c·Y_L$ and $Y_L$ is the uniform vector, $H\left(Z_L\right)\le H\left(Y_L\right)$. It follows that $H\left(Z_L\right)+H\left(Z_R\right)\le$$H\left(Y_L\right)+H\left(Y_R\right)$.

• $s\left(Z_L\right)<s\left(Y_L\right),s\left(Z_R\right)>s\left(Y_R\right)$: there exists a subset $\left(z_{i_1},\dots ,z_{i_{\ell }}\right)$$\subseteq Z_R$ for which every $z_{i_j}$ is greater than the corresponding value $y_{i_j}$ of $Y_R$. Let ${\delta }_{i_j}=z_{i_j}-y_{i_j}$. For every ${\delta }_{i_j}$, there exists a value $z_{\ell }$ in $Z_L$ s.t $z_{\ell }<y_{i_j}<z_{i_j}$, since $s\left(Z_L\right)<s\left(Y_L\right)=c·Y_L<y_{i_j}·Y_L$. Let $Z^{\prime }$ be Z after ${\delta }_{i_j}$ is subtracted from each $z_{i_j}$ and added to some $z_{\ell }\in Z_L$ as described above. By Lemma 1, $H\left(Z^{\prime }\right)>H\left(Z\right)$. It also holds that $Z_R^{\prime }=Y_R$, and that $H\left(Z_L^{\prime }\right)\le H\left(Y_L\right)$, since they both sum to $c·Y_L$, and $Y_L$ is a uniform vector (whose entropy is maximal). Therefore, $H\left(Z\right)<H\left(Z^{\prime }\right)\le H\left(Y\right)$.

• $s\left(Z_L\right)>s\left(Y_L\right),s\left(Z_R\right)<s\left(Y_R\right)$: this case can be immediately omitted; it is an impossibility to feasibly subtract a value from $Z_R$ or increase the value of $s\left(Z_L\right)$ above $s\left(Y_L\right)$.

Therefore, we have proven for any vector Z, it holds that $H\left(Z\right)\le H\left(Y\right)$.    □

Next, we suggest a more time-efficient algorithm than Algorithm 1 that achieves an equivalent bound, with a runtime of $O\left(nlogn\right)$. Suppose c is the maximal threshold all values of the local vectors can be incremented to without exceeding the sum of the values from the remote vector, m. Then, if we define the sorted coordinates of $X^{*}$ to be $x_1,\dots ,x_{n^{\prime }}$, there is some coordinate t such that $x_t\le c\le x_{t+1}$.

By performing a binary search on the the coordinate t of the local vector as described above, we can efficiently find that $x_t$ as described in the algorithm below.

Theorem 2. 

Algorithm 2 runs in $O(nlogn)$ time and returns an upper bound for the entropy of $\tilde{X}=\frac{1}{2}\left(X_{\mathit{Local}}+X_{\mathit{Other}}\right)$.

Algorithm 2: Binary Search Upper Entropy Bound for Two Nodes

Proof. 

The algorithm begins by sorting $X^{*}$, which costs $O\left(nlogn\right)$ and follows by performing a binary search on a range of size $n^{\prime }$, wherein a single step requires $O\left(n\right)$ operations. Therefore, its runtime is $O\left(nlogn\right)$.

Since the vector $X^{*}$ constructed by this algorithm is equivalent to the vector which Algorithm 1 computes, the proof of correctness is the same as the proof of Theorem 1.    □

It will be noted that the upper bound given by Algorithms 1 and 2 can be further improved by using the feasibility constraint upon $X^{*}$. It is possible to increase a coordinate of $X^{*}$ by a value larger than $min\left\{S_K\left(X_{\mathrm{Other}}\right)\right\}$, particularly if for some coordinate i, the inequality $x_{i+1}^{*}-x_i^{*}>min\left\{S_K\left(X_{\mathrm{Other}}\right)\right\}$ holds. In order to keep the core algorithms simple, we will address this formally in Appendix A by proposing an improvement to the algorithms above, such that the bound will indeed by tight.

3.3. Lower Bound

We now turn to discuss a communication-efficient solution for computing a tight lower bound for the entropy of a global vector. As with the upper bound, this problem is an instance of constrained optimization, only that here, our target is to find the minimum. As with the Upper Bound (Section 3.2), we use the same message containing $C_k\left(X\right)$ and $S_k\left(X\right)$, for a remote vector X.

Property 1. 

Let $n^{\prime }$ denote $X^{*}$, $sup$ denote $min\left\{S_k\left(X_{\mathit{Other}}\right)\right\}$ and m denote $1-{\sum }_{x\in S_k\left(X_{\mathit{Other}}\right)}x$, as used in Algorithm 3. Then, $n^{\prime }·sup\ge m$.

Algorithm 3: Lower Entropy Bound for Two Nodes

Proof. 

Using the definitions, we obtain the following inequality:

$$X^{*}·min\left\{S_k\left(X_{\mathrm{Other}}\right)\right\}\ge 1-\sum _{x\in S_k\left(X_{\mathrm{Other}}\right)}x$$

which holds, since:

$$\sum _{x\in S_k\left(X_{\mathrm{Other}}\right)}x+X^{*}·min\left\{S_k\left(X_{\mathrm{Other}}\right)\right\}\ge \sum _{x\in X_{\mathrm{Other}}}x=1.$$

   □

Theorem 3. 

Algorithm 3 runs in $O(nlogn)$ time and returns a tight, lower bound for the entropy of $\tilde{X}=\frac{1}{2}\left(X_{\mathit{Local}}+X_{\mathit{Other}}\right)$

Proof. 

After sorting the vector, we iteratively increment no more than $n^{\prime }=X^{*}\le n$ coordinates since $n^{\prime }·sup\ge m$ by Property 1; hence, the total runtime is $O\left(nlogn\right)$.

To prove the correctness of the bound, it suffices to examine our loop step; it is clear we must add a total sum of m to any of $X^{*}$’s coordinates, and we cannot increment a single coordinate by more than $sup$—since we know all remaining values of the unknown vector X are lesser or equal to $sup$.

The algorithm increments the maximal values of $X^{*}$ by $sup$, which by Corollary 1 incurs the minimal entropy gain to $X^{*}$. Due to the fact that the entropy is coordinate-wise additive, the ”greedy” approach which minimizes over coordinates separately reaches the global minimum.    □

In Figure 1, the proposed algorithm, and the bounds computed using the algorithms described herein, are compared to the bounds derived after sending a random subset of coordinate–value pairs, as well as sending many random subsets and choosing the minimal resulting bound. In Section 3.5, more extensive experiments are reported.

Figure 1.

Comparison of three upper bounds as a function of the number of values sent. The red plot corresponds to an average of random selections of values from the remote vector; the green plot represents the best results (i.e., lower upper bounds) from 10,000 random selections; the blue plot represents the algorithm described here, where $S_k\left(X\right)$ consists of the largest values. The vectors have a length of 100, and their values are sampled from a half-normal distribution with standard deviation 0.02 and then normalized to sum 1.

3.4. Multiparty Bounds

When considering the scale and variability of modern distributed systems, an algorithm that supports multiple machines and incurs a low communication overhead is desirable.

We next suggest a few modifications in order to generalize Algorithms 1–3, for the upper and lower bounds of entropy centralized across $t+1$ nodes. We denote $X_i$ as the vector of machine i, and in a manner similar to Section 3.2, $S_k\left(X_i\right),C_k\left(X_i\right)$ are the ordered sets of the k maximum values and their coordinates, respectively. Typically, the coordinates of $C_k\left(X_i\right)$ and $C_k\left(X_j\right)$ will be disjoint, in which case each machine will have to broadcast its missing coordinates. The additional communication may cost us up to $tk(b+{log}_{2}n)$ bits. We hereby assume a second round of communication occurs, and that $\left(S_k\left(X_1\right),\dots ,S_K\left(X_t\right)\right)$, as well as $\left(C_k\left(X_1\right),\dots ,C_K\left(X_t\right)\right)$ include the same coordinates.

Below, we list the modifications to be made to the previous algorithms for the multiparty case. These changes are similar for all three algorithms.

1. Input: In addition to the local vector $X_{Local}$, the k-sized largest value sets $S_k\left(X_1\right),\dots S_k\left(X_t\right)$ and corresponding ordered coordinate sets $C_k\left(X_1\right),\dots ,C_k\left(X_t\right)$—instead of single sets.

2. The sum to be added to all coordinates of $X^{*}$, m will be $t-{\sum }_{i=1}^t\left({\sum }_{x\in S_k\left(X_i\right)}x\right)$, since there are t local vectors to process and construct, while local vector $X_i$ contributes $1-{\sum }_{x\in X_i}x$.

3. The return value is now $H\left(\frac{1}{t+1}{\tilde{X}}^{\prime }\right)$, since we have summed t additional vectors into $X^{*}$ and $X_{Known}$.

3.5. Experimental Results

To evaluate our algorithms, we tested them on both real and synthetic probability vectors. We now describe the methods and data used to perform our experiments and analyze the results.

In Figure 2 and Figure 3, we simulated the upper bound algorithm (Algorithm 2) and the lower bound algorithm (Algorithm 3). Figure 2b depicts a simulation of the algorithms on two randomly generated vectors: $Nod{e}_1$ with uniform distribution and $Nod{e}_2$ with beta distribution. The probability vectors of $Nod{e}_1$ and $Nod{e}_2$ are shown in Figure 2a. Note that as depicted in Figure 2b, the algorithms’ results in each node are determined by the distribution of the local probability vectors. That is because the more probability mass is transmitted, the tighter the bounds become, and the quicker it converges with respect to k. As illustrated, in the bounds of $Nod{e}_1$, which receives $Nod{e}_2$’s maximal beta distribution’s probability values, the bounds converge quickly with respect to k. In contrast, the bounds that are computed at $Nod{e}_2$, which receives the maximal values of the uniform distribution of $Nod{e}_1$, converge slowly to the real entropy. This is due to the fact $Nod{e}_2$ does not gain much information from $Nod{e}_1$.

Figure 2.

Algorithmic bounds for the empirical entropy on synthetic probability vector of dimension 50 k. (a) depicts the distributions of the generated vectors: a normalized uniform distribution (i.e., each value is randomly selected from $U[0,1]$, and then their sum is normalized to 1) which for brevity we refer to as “uniform” at $Nod{e}_1$, and a beta distribution at $Nod{e}_2$ with parameters $\alpha =0.2,\beta =100$. The dashed line is the average vector of the two. (b) demonstrates the locally calculated upper bound and lower bound for different numbers of top values transmitted (k) in Algorithms 2 and 3.

Figure 3.

Algorithmic bounds between token frequency vectors of atheism-themed newsgroups and hockey-themed newsgroups. (a) depicts the histogram of the tokens of the accumulation of the first 200 articles in each theme. Note that the histogram’s coordinates are organized in descending order of each vector’s values separately; thus, the average vector may be larger, for some values, from both $Nod{e}_1$ and $Nod{e}_2$. (b) demonstrates the locally computed upper bound and lower bound for different numbers of top values transmitted (k) in Algorithms 2 and 3.

Fortunately, the difference between the bounds of $Nod{e}_1$ and $Nod{e}_2$ is an advantage to our proposed algorithms; we can compare them and use the better one simply by comparing the bounds (which requires transmitting only one scalar).

Another interesting observation can be drawn from Figure 2b; $Nod{e}_1$’s lower bound is already quite close to the global entropy for very small k values. The algorithm works well here since the maximum value of $Nod{e}_2$ is not large, which in turn enables the algorithm to reach a tighter bound.

Figure 3b illustrates our experiments on the 20 Newsgroups Dataset [13] which includes about 20,000 newsgroup documents for 20 different topics. We measured the entropy of token frequency vectors (A vector where each value corresponds to the frequency of a word or token in the document).  from the atheism-themed newsgroups and the hockey-themed newsgroups. To do so, we took the top 10,000 occurring tokens and created token frequency vectors on the first 200 articles from the atheism theme and the hockey theme. The visual illustration of the (sorted) tokens frequency is in Figure 3a. As can be observed, the atheism newsgroups is more verbally rich than the hockey newsgroup, having more words which are unique to it.

As demonstrated in Figure 3b, the upper bound computed by both nodes is almost the same. However, for the lower bound, $Nod{e}_1$ (atheism) converges faster to the real entropy as we increase the parameter k of the algorithm. We attribute that to the denser token histogram of the hockey theme; thus, more “probability mass” is transmitted for the same k.

Figure 4 presents results for the multiparty case, as discussed in Section 3.4.

Figure 4.

An example of the multiparty upper bound on real and synthetic data. (a) For the real data, we used vectors from the newsgroups dataset; each vector is of length 10,000. (b) The synthetic data are sampled from half-normal distribution; each vector is of length 1000.

To conclude, the distribution of the probability vectors directly affects the tightness of the bounds. The less concentrated the probability vectors are, the less information we can send for every k; hence, the bounds become less tight, as demonstrated in Figure 5. A solution for this case is presented in Section 4.

Figure 5.

Rate of convergence of the dynamic bound algorithms in Section 3 to the real entropy values as a function of communication overhead. (a) $Nod{e}_2$ obeys a uniform distribution, and $Nod{e}_1$ obeys a beta distribution with $\alpha =0.1$, $\beta =100$. (b) Both nodes obey a beta distribution, one with $\alpha =0.1$, $\beta =100$ and the other with $\alpha =0.02$, $\beta =100$.

4. Entropy Approximation

The algorithms described in Section 3 perform better in terms of communication overhead when there are a few relatively large values in the local frequency vectors, i.e., a substantial percentage of the overall “probability mass” resides in a relatively small percentage of the vectors’ values. However, in the case in which the vectors are “flat”—that is, their distribution approaches a uniform one—the nodes will have to exchange many values in order to reach tight bounds on the overall entropy; see Figure 5. In this section, we offer a probabilistic solution to this problem.

Assume that two nodes $N_1,N_2$ hold vectors $X,Y$, and the goal is to approximate the entropy of the average vector $\frac{X+Y}{2}$, with a small communication overhead, relative to n, the length of the vectors.

One solution, which was applied in previous work on monitoring entropy [10], is to use sketches. This popular technique found many applications in computer science, for example, for computations over distributed data [7]. A well-known sketch for entropy, which we describe in Section 4.1, is presented in [14]; see also [15].

Here we use a different sketch, which, for our purposes, performed better than the sketch presented in [14]. In resemblance to Section 3, the two nodes first exchange all values which are greater or equal to a threshold $\epsilon$, whose value is determined by a communication/accuracy trade-off. Hence, we assume hereafter that all values are smaller than $\epsilon$. Next, choose a polynomial approximation, of degree at least 2, over the interval $[0,\epsilon ]$, to the function $h\left(t\right)\triangleq -tln\left(t\right)$. Assuming in the meanwhile a degree 2 approximation, denote it by $A{t}^2+Bt+C$. The proposed method is oblivious to the choice of this approximation; we have used the approach of minimizing

$${\int }_0^{\epsilon }{\left(f\left(t\right)-(A{t}^2+Bt+C)\right)}^{2}dt,$$

which allows a closed-form solution

$$A=-\frac{5}{4\epsilon },B=-ln\left(\epsilon \right)+\frac{13}{12},C=\frac{\epsilon }{8}.$$

Using this quadratic function, we can approximate the entropy of the average vector $\frac{X+Y}{2}$ by

$$\begin{array}{cc}\hfill & \sum _{i=1}^n\left(A{\left(\frac{X_i+Y_i}{2}\right)}^2+B\left(\frac{X_i+Y_i}{2}\right)+C\right)=\hfill \\ \hfill \frac{A}{4}& \sum _{i=1}^n(X_i^2+Y_i^2)+\frac{A}{2}\sum _{i=1}^n{X}_i{Y}_i+\frac{B}{2}\sum _{i=1}^n(X_i+Y_i)+nC.\hfill \end{array}$$

Note that with the exception of the term ${\sum }_{i=1}^n{X}_i{Y}_i$, all terms can be computed locally and require $O\left(1\right)$ communication overhead to transmit. Thus, it only remains to approximate the term ${\sum }_i{X}_i{Y}_i$, which equals the inner product $\langle X,Y\rangle$. To this end, we can apply an approximation based on the famed Johnson–Lindenstrauss Lemma [16], which is defined as follows:

$$\langle X,Y\rangle \approx \frac{1}{d}\sum _{i=1}^d\langle X,R_i\rangle \langle Y,R_i\rangle ;$$

where $R_i$ are independent random vectors with all values i.i.d standard normal variables, which are generated by a pre-agreed upon random seed, and thus require no communication. A direct calculation yields that this estimate has expectation $\langle X,Y\rangle$ (i.e., is unbiased) and its variance equals

$$\frac{{\parallel X\parallel }^2{\parallel Y\parallel }^2+\langle X,Y\rangle }{d}.$$

Similarly, we can apply higher-order approximations. For a cubic approximation, we obtain a more complicated but identical in spirit sketch, which requires an approximation of the expressions ${\sum }_{i=1}^n{X}_i^2{Y}_i,{\sum }_{i=1}^n{X}_i{Y}_i^2$; this, too, can be achieved by applying the estimate above, since these quantities can also be represented as inner products of “local vectors”: for example, ${\sum }_{i=1}^n{X}_i^2{Y}_i=\langle X^2,Y\rangle$, where ${\left(X^2\right)}_i\triangleq X_i^2$.

Some results for two nodes are presented in Figure 6, in which the proposed sketch is compared to the one in [14] (see Section 4.1). Extending the sketch to the multiparty scenario is straightforward; results are presented in Figure 7b.

Figure 6.

Comparison of the Poly2 and CC sketches for approximating the empirical Shannon entropy. (a) illustrates the synthetic probability vectors that were generated to perform the comparison. (b) compares the Poly2 sketch to the CC sketch for varying sketch sizes. The comparison was made using three different random seeds for the sketches. We used the value $\epsilon =0.0002$.

Figure 7.

(a): standard deviation of the error of the CC and Poly2 sketches, for two parties and varying sketch size. The experiments were performed on a vector of dimension 10000 with uniform distribution, which was followed by normalization to sum 1. The standard deviation was calculated on 50 sketches for each sketch size. (b): comparison of the standard deviation of CC and Poly2 sketches in the multiparty scenario for fixed sketch size and varying number of parties. The experiments were performed for an i.i.d random vector distribution of dimension 5000 with sketch size 200 and $\epsilon =0.0002$.

4.1. The Clifford-Cosma Sketch

We compare our sketch to an entropy sketch proposed in  [14]. The sketch is a linear projection of the probability vector. The linear projection is performed by a multiplication matrix with i.i.d elements drawn from $F(x;1,-1,\pi /2,0)$. The entropy approximation of the d-dimensional linear projected vector $(y_1,\dots ,y_d)$ is:

$$\tilde{H}(y_1,\dots ,y_d)=ln\left(d\right)-ln\left(\sum _{i=1}^d{e}^{y_i}\right).$$

4.2. Sketch Evaluation

We now compare the proposed sketch to the one in [14], which is denoted “CC”. The proposed quadratic sketch is denoted “Poly2”.

5. Conclusions

We have presented novel communication-efficient algorithms for bounding and approximating the entropy in a distributed setting. The algorithms were tested on real and synthetic data, yielding a substantial reduction in communication overhead. Future work will address both sketch-based techniques and further development of the dynamic bound algorithms presented here. In addition, we intend to address the efficient distributed computation of other functions.

Appendix A. Improving the Upper Bound

Let us recall Algorithms 1 and 2 from Section 3.2. As mentioned there, the computed bound can be improved; we now present this improvement as a generalized algorithm that achieves a tight upper bound. The algorithm below can be run following the upper bound algorithms (either of them), after $X^{*}$ is computed, or as an external procedure that accepts $X^{*}$, m and $sup$ as input. We show the former option:

Algorithm A1: Tight Entropy Upper Bound for Two Nodes

Author Contributions

Conceptualization, A.S., Y.A. and D.K; methodology, A.S., Y.A. and D.K.; software, A.S. and Y.A.; validation, A.S., Y.A. and D.K.; formal analysis, A.S., Y.A. and D.K.; investigation, A.S., Y.A. and D.K.; resources, A.S., Y.A. and D.K.; data curation, A.S., Y.A. and D.K.; writing—original draft preparation, A.S., Y.A. and D.K.; writing—review and editing, A.S., Y.A. and D.K.; visualization, A.S., Y.A. and D.K.; supervision, D.K.; project administration, D.K. All authors have read and agreed to the published version of the manuscript.

Data Availability Statement

The 20 Newsgroups Dataset, which can be found in this link.

Conflicts of Interest

The authors declare no conflict of interest.

Funding Statement

This research received no external funding.