This tutorial paper focuses on the variants of the bottleneck problem taking an information theoretic perspective and discusses practical methods to solve it, as well as its connection to coding and learning aspects. The intimate connections of this setting to remote source-coding under logarithmic loss distortion measure, information combining, common reconstruction, the Wyner–Ahlswede–Korner problem, the efficiency of investment information, as well as, generalization, variational inference, representation learning, autoencoders, and others are highlighted. We discuss its extension to the distributed information bottleneck problem with emphasis on the Gaussian model and highlight the basic connections to the uplink Cloud Radio Access Networks (CRAN) with oblivious processing. For this model, the optimal trade-offs between relevance (i.e., information) and complexity (i.e., rates) in the discrete and vector Gaussian frameworks is determined. In the concluding outlook, some interesting problems are mentioned such as the characterization of the optimal inputs (“features”) distributions under power limitations maximizing the “relevance” for the Gaussian information bottleneck, under “complexity” constraints.

The problem of distributed representation learning is one in which multiple observations $X_1,\ldots,X_K$ are processed separately to learn as much information as possible about some source $Y$ . We investigate this problem from information-theoretic grounds, through a generalization of Tishby's centralized Information Bottleneck (IB) method to the distributed setting. Specifically, $K\geq 2$ encoders, compress their observations $X_1,\ldots,X_K$ separately such that, collectively, the produced representations preserve as much information as possible about $Y$ . We study both discrete memoryless (DM) and vector Gaussian data models. For the discrete model, we establish a single-letter characterization of the optimal tradeoff for a class of memoryless sources. For the vector Gaussian model, we provide an explicit characterization of the optimal complexity-relevance tradeoff. Furthermore, we develop a variational bound on the complexity-relevance tradeoff which generalizes the evidence lower bound (ELBO) to the distributed setting. We provide two algorithms to compute this bound: i) a Blahut-Arimoto type iterative algorithm which computes optimal complexity-relevance mappings by iterating over a set of self-consistent equations, and ii) a variational inference type algorithm in which the encoding mappings are parametrized by neural networks, the bound approximated by Markov sampling and optimized with stochastic gradient descent. Numerical results on synthetic and real datasets are provided to support the efficiency of the approaches and algorithms developed in this paper.

In this paper, we develop an unsupervised generative clustering framework that combines the variational information bottleneck and the Gaussian mixture model. Specifically, in our approach, we use the variational information bottleneck method and model the latent space as a mixture of Gaussians. We derive a bound on the cost function of our model that generalizes the Evidence Lower Bound (ELBO) and provide a variational inference type algorithm that allows computing it. In the algorithm, the coders’ mappings are parametrized using neural networks, and the bound is approximated by Markov sampling and optimized with stochastic gradient descent. Numerical results on real datasets are provided to support the efficiency of our method.

A single-sensor two-detectors system is considered where the sensor communicates with both detectors and Detector 1 communicates with Detector 2, all over noise-free rate-limited links. The sensor and both detectors observe discrete memoryless source sequences whose joint probability mass function depends on a binary hypothesis. The goal at each detector is to guess the binary hypothesis in a way that, for increasing observation lengths, the probability of error under one of the hypotheses decays to zero with largest possible exponential decay, whereas the probability of error under the other hypothesis can decay to zero or to a small positive number arbitrarily slow. For the setting with positive communication rates from the sensor to the detectors and when both detectors are interested in maximizing the error exponent under the same hypothesis, we characterize the set of all possible exponents in a special case of testing against independence. In this case the cooperation link allows Detector 2 to increase its Type-II error exponent by an amount that is equal to the exponent attained at Detector 1. We also provide a general inner bound on the set of achievable error exponents that shows a tradeoff between the exponents at the two detectors in most cases. When the two detectors aim at maximizing the error exponent under different hypotheses and the distribution at the Sensor is different under the two hypotheses, then we show that such a tradeoff does not exist. We propose a general scheme that allows each detector to attain the same exponent as if it was the only detector in the system. For the setting with zero-rate communication on both links, we exactly characterize the set of possible exponents and the gain brought up by cooperation, in function of the number of bits that are sent over the two links. Notice that, for this setting, tradeoffs between the exponents achieved at the two detectors arise only in few particular cases. In all other cases, each detector achieves the same performance as if it were the only detector in the system.

Cooperative information embedding deals with the problem of embedding unperceived information into some cover signal by different users or partners, cooperatively. It models applications in which embedded signals, or watermarks, transmitted over wireless networks need to be reinforced in order to withstand channel impairments. In cooperative information embedding, each embedder that can reinforce the embedded signal may or may not know the original cover signal. In this paper, we concentrate on the two user cases: (1) an initial embedder and (2) an assisting embedder or helper collaborate to embed some watermark into given digital media content which is transmitted over a wireless network. One important application is that of infrastructure-aided information embedding, a case in which the network provider plays the role of a helper and contributes to securing the distribution of the media, not only by blocking unauthorized signals but also by reinforcing the watermarks in legitimate signals. We investigate the two scenarios in which the helper does or does not know the cover signal. For each scenario, we derive lower and upper bounds on channel capacity. Furthermore, we also design implementable coding schemes and derive the embedding rates practically allowed by these schemes, for both scenarios. Among others, the performance characterization shows that for cooperative information embedding to be effective, careful code design is required at the initial embedder and the helper. The careful design concerns the joint conception of the embedded codes and the exploitation of the knowledge of the cover signal, if any.