31 January 2025 in TechRxiv
The Information Bottleneck theory offers a framework for understanding the learning dynamics of deep neural networks (DNNs) by analyzing the mutual information (MI) between inputs X and the network's feature representations T , I(X; T), and between T and outputs Y , I(T ; Y), on a plane. Despite its theoretical appeal, some limitations exist: for deterministic networks, the true I(X; T) is infinite, and estimating MI in high-dimensional spaces is notoriously difficult, often resulting in inconsistent empirical findings. In this work, we introduce the Sliced Information Plane (SIP), using Sliced Mutual Information (SMI) as an alternative to MI. The reason for utilizing SMI is that it is scalable to high dimensions and remains finite even with deterministic mappings. In our study, we visualize the SIP which captures the relationship between SI(X; T) (SMI between X and T) and SI(T ; Y) (SMI between T and Y). We show that the trends are consistent across different estimators and we utilize SIP to analyze the learning dynamics of DNNs under various settings. An interesting observation is during training, SI(X; T) initially decreases and then increases for deeper layers, suggesting that the network first discards what it deems irrelevant information from the input, only to later relearn some of this information, which it considers useful for predicting the label (increase in SI(T ; Y)).