The empirical risk minimization (ERM) problem with relative entropy regularization (ERM-RER) is investigated under the assumption that the reference measure is a $\sigma $ -finite measure, and not necessarily a probability measure. Under this assumption, which leads to a generalization of the ERM-RER problem allowing a larger degree of flexibility for incorporating prior knowledge, numerous relevant properties are stated. Among these properties, the solution to this problem, if it exists, is shown to be a unique probability measure, mutually absolutely continuous with the reference measure. Such a solution exhibits a probably-approximately-correct guarantee for the ERM problem independently of whether the latter possesses a solution. For a fixed dataset and under a specific condition, the empirical risk is shown to be a sub-Gaussian random variable when the models are sampled from the solution to the ERM-RER problem. The generalization capabilities of the solution to the ERM-RER problem (the Gibbs algorithm) are studied via the sensitivity of the expected empirical risk to deviations from such a solution towards alternative probability measures. Finally, an interesting connection between sensitivity, generalization error, and lautum information is established.
6 October 2021 in IEEE Journal on Selected Areas in Communications
In decentralized optimization, multiple nodes in a network collaborate to minimize the sum of their local loss functions. The information exchange between nodes required for this task, is often limited by network connectivity. We consider a setting in which communication between nodes is hindered by both (i) a finite rate-constraint on the signal transmitted by any node, and (ii) additive noise corrupting the signal received by any node. We propose a novel algorithm for this scenario: Decentralized Lazy Mirror Descent with Differential Exchanges (DLMD-DiffEx), which guarantees convergence of the local estimates to the optimal solution under the given communication constraints. A salient feature of DLMD-DiffEx is the introduction of additional proxy variables that are maintained by the nodes to account for the disagreement in their estimates due to channel noise and rate-constraints. Convergence to the optimal solution is attained by having nodes iteratively exchange these disagreement terms until consensus is achieved. In order to prevent noise accumulation during this exchange, DLMD-DiffEx relies on two sequences: one controlling the power of the transmitted signal, and the other determining the consensus rate. We provide insights on the design of these two sequences which highlights the interplay between consensus rate and noise amplification. We investigate the performance of DLMD-DiffEx both from a theoretical perspective as well as through numerical evaluations on synthetic data and MNIST. MATLAB and Python implementations can be found at https://github.com/rajarshisaha95/DLMD-DiffEx.
13 June 2022 in IEEE Transactions on Signal Processing
Unsourced random access (URA) is a recently proposed multiple access paradigm tailored to the uplink channel of machine-type communication networks. By exploiting a strong connection between URA and compressed sensing, the massive multiple access problem may be cast as a compressed sensing (CS) problem, albeit one in exceedingly large dimensions. To efficiently handle the dimensionality of the problem, coded compressed sensing (CCS) has emerged as a pragmatic signal processing tool that, when applied to URA, offers good performance at low complexity. While CCS is effective at recovering a signal that is sparse with respect to a single basis, it is unable to jointly recover signals that are sparse with respect to separate bases. In this article, the CCS framework is extended to the demixing setting, yielding a novel technique called coded demixing. A generalized framework for coded demixing is presented and a low-complexity recovery algorithm based on approximate message passing (AMP) is developed. Coded demixing is applied to heterogeneous multi-class URA networks and traditional single-class networks. Its performance is analyzed and numerical simulations are presented to highlight the benefits of coded demixing.
26 March 2021 in IEEE Transactions on Information Theory
Consider the problem of estimating a latent signal from a lossy compressed version of the data when the compressor is agnostic to the relation between the signal and the data. This situation arises in a host of modern applications when data is transmitted or stored prior to determining the downstream inference task. Given a bitrate constraint and a distortion measure between the data and its compressed version, let us consider the joint distribution achieving Shannon's rate-distortion (RD) function. Given an estimator and a loss function associated with the downstream inference task, define the RD risk as the expected loss under the RD-achieving distribution. We provide general conditions under which the operational risk in estimating from the compressed data is asymptotically equivalent to the RD risk. The main theoretical tools to prove this equivalence are transportation-cost inequalities in conjunction with properties of compression codes achieving Shannon's RD function. Whenever such equivalence holds, a recipe for designing estimators from datasets undergoing lossy compression without specifying the actual compression technique emerges: design the estimator to minimize the RD risk. Our conditions are simplified in the special cases of discrete memoryless or multivariate normal data. For these scenarios, we derive explicit expressions for the RD risk of several estimators and compare them to the optimal source coding performance associated with full knowledge of the relation between the latent signal and the data.
16 March 2023 in IEEE Transactions on Communications
Recently, several methods have been proposed for estimating the mutual information from sample data using deep neural networks. This approach is referred to as neural mutual information estimation (NMIE). NMIEs differ from other approaches in the literature as they are data-driven estimators. As such, they have the potential to perform well on a large class of capacity problems. To test the performance across various NMIEs, it is desirable to establish a benchmark encompassing the different challenges of capacity estimation. This is the objective of this paper. We consider three scenarios for benchmarking: (i) the classic AWGN channel, (ii) channels continuous inputs – the optical intensity and peak-power constrained AWGN channel (iii) channels with a discrete output – i.e., the Poisson channel. We also consider the extension to the multi-terminal case with (iv) the AWGN and optical MAC models. We argue that benchmarking a certain NMIE across these four scenarios provides a substantive test of performance. We study the performance of mutual information neural estimator (MINE), smoothed mutual information lower-bound estimator (SMILE), and directed information neural estimator (DINE) and provide insights into the performance of other methods as well. To summarize our benchmarking results, MINE provides the most reliable performance.
24 June 2019 in IEEE Transactions on Wireless Communications
We consider the multiterminal remote source coding problem of estimating a Gaussian signal from a bit-restricted representation of distributed linear measurements corrupted by additive white Gaussian noise. For this problem, we study the performance of the multiterminal compress-and-estimate (CE) coding scheme in which multiple remote encoders compress their measurements so as to minimize a local distortion measure which depends solely on the distribution of these measurements. In reconstruction, the decoder estimates the signal from the lossy-compressed measurements having full knowledge of the statistics of the source signal and the noisy measurements. The CE coding scheme is motivated by the scenario in which source encoders, due to their limited capabilities, operate according to a pre-determined compression strategy and cannot adapt to the sensing environment while the fusion center has full knowledge and computational capabilities. We focus, in particular, on two scenarios: the centralized observation model in which measurements are collected at a single remote encoder and the distributed observation model where measurements are provided to multiple remote sensors. In both scenarios, we investigate the performance attainable through the CE coding scheme in which the measurements are compressed according to a quadratic distortion measure and compare it to the performance of the coding scheme having full system knowledge.