It turns out to be a favourable week or two for me to finally finish a number of papers that had been at a nearly completed stage for a while. I have just uploaded to the arXiv my article “Sumset and inverse sumset theorems for Shannon entropy“, submitted to Combinatorics, Probability, and Computing. This paper evolved from a “deleted scene” in my book with Van Vu entitled “Entropy sumset estimates“. In those notes, we developed analogues of the standard Plünnecke-Ruzsa sumset estimates (which relate quantities such as the cardinalities of the sum and difference sets of two finite sets in an additive group to each other), to the entropy setting, in which the finite sets are replaced instead with discrete random variables taking values in that group G, and the (logarithm of the) cardinality |A| is replaced with the Shannon entropy
This quantity measures the information content of X; for instance, if , then it will take k bits on the average to store the value of X (thus a string of n independent copies of X will require about nk bits of storage in the asymptotic limit ). The relationship between entropy and cardinality is that if X is the uniform distribution on a finite non-empty set A, then . If instead X is non-uniformly distributed on A, one has , thanks to Jensen’s inequality.
It turns out that many estimates on sumsets have entropy analogues, which resemble the “logarithm” of the sumset estimates. For instance, the trivial bounds
have the entropy analogue
whenever X, Y are independent discrete random variables in an additive group; this is not difficult to deduce from standard entropy inequalities. Slightly more non-trivially, the sum set estimate
established by Ruzsa, has an entropy analogue
,
and similarly for a number of other standard sumset inequalities in the literature (e.g. the Rusza triangle inequality, the Plünnecke-Rusza inequality, and the Balog-Szemeredi-Gowers theorem, though the entropy analogue of the latter requires a little bit of care to state). These inequalities can actually be deduced fairly easily from elementary arithmetic identities, together with standard entropy inequalities, most notably the submodularity inequality
whenever X,Y,Z,W are discrete random variables such that X and Y each determine W separately (thus for some deterministic functions f, g) and X and Y determine Z jointly (thus for some deterministic function f). For instance, if X,Y,Z are independent discrete random variables in an additive group G, then and each determine separately, and determine jointly, leading to the inequality
which soon leads to the entropy Rusza triangle inequality
which is an analogue of the combinatorial Ruzsa triangle inequality
All of this was already in the unpublished notes with Van, though I include it in this paper in order to place it in the literature. The main novelty of the paper, though, is to consider the entropy analogue of Freiman’s theorem, which classifies those sets A for which . Here, the analogous problem is to classify the random variables such that , where are independent copies of X. Let us say that X has small doubling if this is the case.
For instance, the uniform distribution U on a finite subgroup H of G has small doubling (in fact in this case). In a similar spirit, the uniform distribution on a (generalised) arithmetic progression P also has small doubling, as does the uniform distribution on a coset progression H+P. Also, if X has small doubling, and Y has bounded entropy, then X+Y also has small doubling, even if Y and X are not independent. The main theorem is that these are the only cases:
Theorem 1. (Informal statement) X has small doubling if and only if for some uniform distribution U on a coset progression (of bounded rank), and Y has bounded entropy.
For instance, suppose that X was the uniform distribution on a dense subset A of a finite group G. Then Theorem 1 asserts that X is close in a “transport metric” sense to the uniform distribution U on G, in the sense that it is possible to rearrange or transport the probability distribution of X to the probability distribution of U (or vice versa) by shifting each component of the mass of X by an amount Y which has bounded entropy (which basically means that it primarily ranges inside a set of bounded cardinality). The way one shows this is by randomly translating the mass of X around by a few random shifts to approximately uniformise the distribution, and then deal with the residual fluctuation in the distribution by hand. Theorem 1 as a whole is established by using the Freiman theorem in the combinatorial setting combined with various elementary convexity and entropy inequality arguments to reduce matters to the above model case when X is supported inside a finite group G and has near-maximal entropy.
I also show a variant of the above statement: if X, Y are independent and , then we have (i.e. X has the same distribution as Y+Z for some Z of bounded entropy (not necessarily independent of X or Y). Thus if two random variables are additively related to each other, then they can be additively transported to each other by using a bounded amount of entropy.
In the last part of the paper I relate these discrete entropies to their continuous counterparts
where X is now a continuous random variable on the real line with density function . There are a number of sum set inequalities known in this setting, for instance
,
for independent copies of a finite entropy random variable X, with equality if and only if X is a Gaussian. Using this inequality and Theorem 1, I show a discrete version, namely that
,
whenever and are independent copies of a random variable in (or any other torsion-free abelian group) whose entropy is sufficiently large depending on . This is somewhat analogous to the classical sumset inequality
though notice that we have a gain of just rather than here, the point being that there is a Gaussian counterexample in the entropy setting which does not have a combinatorial analogue (except perhaps in the high-dimensional limit). The main idea is to use Theorem 1 to trap most of X inside a coset progression, at which point one can use Fourier-analytic additive combinatorial tools to show that the distribution is “smooth” in some non-trivial direction r, which can then be used to approximate the discrete distribution by a continuous one.
I also conjecture more generally that the entropy monotonicity inequalities established by Artstein, Barthe, Ball, and Naor in the continuous case also hold in the above sense in the discrete case, though my method of proof breaks down because I no longer can assume small doubling.
11 comments
Comments feed for this article
26 June, 2009 at 2:26 am
maxbaroi
Dear Professor Tao,
“It turns out to be a favourable week or two for me to finally finish a number of papers that had been at a nearly completed stage for a while. ”
I would imagine that summer is a more productive time as you don’t have to put up with your students.
More seriously, does the similarity in form between the estimates involving the discrete Shannon entropy and the sumsets estimates of finite subsets of an additive group hold when you consider their continuous counterparts? Where I’m assuming the continuous analog is when we replace finite subsets of an additive group by Lebesgue measurable sets and instead of considering cardinality, we consider the Lebesgue measure.
Thanks for any reply,
Max
27 June, 2009 at 7:19 am
Terence Tao
Yes, the discrete sumset and entropy results can be used to derive continuous counterparts by discretising the latter at some small spatial scale and then sending that scale to zero.
The converse procedure is less clear though; there are some continuous entropy results (such as those of Artstein, Barthe, Ball and Naor mentioned above) which I was not able to easily convert to the discrete setting. (If one approximates a discrete random variable by a continuous counterpart, by adding a small amount of gaussian noise, this noise expands under addition and will eventually contribute unwanted correction terms such as to entropy sumset inequlaities.)
2 December, 2009 at 3:06 pm
AJ
Professor Tao:
Regarding the continuous counter part to entropy, I have a comment. Continuous entropy is not usually well behaved. It can be negative, is not invariant to scaling, even worse there is no renormalized entropy in the sense that self entropy can be given a value of infinity. Given this could the results be still transferred? I would be interested in the continuous counterpart. Has anyone worked on this?
2 December, 2009 at 3:08 pm
AJ
I apologize I did not read the ending comments on the blog.
But I think scale invariant and normalized continuous entropy would be an useful tool to information theorists.
26 February, 2010 at 2:28 pm
YP
Perhaps one way is to consider Kullback-Leibler divergence D(P||Q), where Q is something nice absolutely-continuous (e.g. N(0,1) gaussian); instead of H_R(P), which is simply a -D(P||Leb). The bad properties of H_R(P) come from the fact that you take divergence w.r.t. to a non-probability measure.
26 June, 2009 at 8:43 am
link
you can easily replace the Shannon entropy with the standard measurement of information in quantum states (a.f., S = ln N).
27 October, 2009 at 8:59 pm
An entropy Plünnecke-Ruzsa inequality « What’s new
[…] a recent paper, I adapted a number of sum set estimates to the entropy setting, in which finite sets such as in […]
16 October, 2010 at 4:07 pm
LifeIsChill
Let $\mathbb N_{n} = \{1,2,\cdots,n\}$.
Let $S$ be of cardinality $n$ where elements of $S$ are integers from $\mathbb N_{n}$ and at least one element of $S$ is repeated (That is at least one integer from $\mathbb N_{n}$ is skipped. One can easily find a set $S$ with the property that:
$\displaystyle \sum_{j \in S}j^{i} = \displaystyle \sum_{j \in \mathbb N_{n}j^{i}$
when $i = 1$.
How about for $i \ge 2$? It is not obvious that higher power sum sets exist due to constraint in the cardinality of $S$ and $\mathbb N_{n}$. One cannot deny it either? Is there a easy way to tackle some sumset questions?
3 September, 2012 at 12:25 am
On the entropy of a sum | MoVn - Linux Ubuntu Center
[…] short, the bound $(*)$ overshoots by quite a bit in this situation. From perusing this blog post, I gather all sorts of bounds on $H(X+Y)$ are possible; is there a bound that gives the right […]
13 September, 2012 at 9:00 pm
On the entropy of a sum | Q&A System
[…] a nutshell, the bound $(*)$ overshoots by a great deal in cases like this. From perusing this web site publish, I gather a variety of bounds on $H(X Y)$ are possible it is possible to bound that provides the […]
27 July, 2018 at 5:59 am
Entropy and Sumsets: An example | George Shakan
[…] of sumset inequalities hold and sometimes they do not (see this paper of Ruzsa or this paper this paper of Tao, or a host of work by Madiman and […]