Inspired by the pioneering work of the information bottleneck (IB) principle for Deep Neural Networks’ (DNNs) analysis, we thoroughly study the relationship among the model accuracy, and , where and are the mutual information of DNN’s output T with input X and label Y. Then, we design an information plane-based framework to evaluate the capability of DNNs (including CNNs) for image classification. Instead of each hidden layer’s output, our framework focuses on the model output T. We successfully apply our framework to many application scenarios arising in deep learning and image classification problems, such as image classification with unbalanced data distribution, model selection, and transfer learning. The experimental results verify the effectiveness of the information plane-based framework: Our framework may facilitate a quick model selection and determine the number of samples needed for each class in the unbalanced classification problem. Furthermore, the framework explains the efficiency of transfer learning in the deep learning area.
14 October 2025 in IEEE Transactions on Information Theory
Bistatic sensing refers to scenarios where the transmitter (illuminating the target) and the sensing receiver (estimating the target state) are physically separated, in contrast to monostatic sensing, where both functions are co-located. In practical settings, bistatic sensing may be required either due to inherent system constraints or as a means to mitigate the strong self-interference encountered in monostatic configurations. A key practical challenge in bistatic radio-frequency radar systems is the synchronization and calibration of the separate transmitter and sensing receiver. In this paper, we are not concerned with these signal processing aspects and take a complementary information-theoretic perspective on bistatic integrated sensing and communication (ISAC). Namely, we aim to characterize the capacity-distortion function—the fundamental tradeoff between communication capacity and sensing accuracy. We consider a general discrete channel model for a bistatic ISAC system and derive a multi-letter representation of its capacity-distortion function. Then, we establish single-letter upper and lower bounds and provide exact single-letter characterizations for degraded bistatic ISAC channels. Numerical examples illustrate the theoretical results, highlighting the benefits of ISAC over separate communication and sensing, as well as the role of leveraging communication to assist sensing in bistatic systems.
Let Z be a standard Gaussian random variable, X be independent of Z, and t be a strictly positive scalar. For the derivatives in t of the differential entropy of , McKean noticed that Gaussian X achieves the extreme for the first and second derivatives, among distributions with a fixed variance, and he conjectured that this holds for general orders of derivatives. This conjecture implies that the signs of the derivatives alternate. Recently, Cheng and Geng proved that this alternation holds for the first four orders. In this work, we employ the technique of linear matrix inequalities to show that: firstly, Cheng and Geng’s method may not generalize to higher orders; secondly, when the probability density function of is log-concave, McKean’s conjecture holds for orders up to at least five. As a corollary, we also recover Toscani’s result on the sign of the third derivative of the entropy power of , using a much simpler argument.
In this paper, for a general full-duplex wireless MapReduce distributed computing network, we investigate the minimization of the communication overhead for a given computation overhead. The wireless MapReduce framework consists of three phases: Map phase, Shuffle phase and Reduce phase. Specifically, we model the Shuffle phase into a cooperative X network based on a more general file assignment strategy. Furthermore, for this cooperative X network, we derive an information-theoretic upper bound on the sum degree of freedom (SDoF). Moreover, we propose a joint interference alignment and neutralization (IAN) scheme to characterize the achievable SDoF. Especially, in some cases, the achievable SDoF coincides with the upper bound on the SDoF, hence, the IAN scheme provides the optimal SDoF. Finally, based on the SDoF, we present an information-theoretic lower bound on the normalized delivery time (NDT) and achievable NDT of the wireless distributed computing network, which are less than or equal to those of the existing networks. The lower bound on the NDT shows that 1) there is a tradeoff between the computation load and the NDT; 2) the achievable NDT is optimal in some cases, hence, the proposed IAN scheme can reduce the communication overhead effectively.
The celebrated Blahut–Arimoto algorithm computes the capacity of a discrete memoryless point-to-point channel by alternately maximizing the objective function of a maximization problem. This algorithm has been applied to degraded broadcast channels, in which the supporting hyperplanes of the capacity region are again cast as maximization problems. In this work, we consider general broadcast channels and extend this algorithm to compute inner and outer bounds on the capacity regions. Our main contributions are as follows: first, we show that the optimization problems are max–min problems and that the exchange of minimum and maximum holds; second, we design Blahut–Arimoto algorithms for the maximization part and gradient descent algorithms for the minimization part; third, we provide convergence analysis for both parts. Numerical experiments validate the effectiveness of our algorithms.