2 February 2021 in IEEE Journal on Selected Areas in Information Theory
We consider a distributed empirical risk minimization (ERM) optimization problem with communication efficiency and privacy requirements, motivated by the federated learning (FL) framework (Kairouz et al., 2019). Unique challenges to the traditional ERM problem in the context of FL include (i) need to provide privacy guarantees on clients' data, (ii) compress the communication between clients and the server, since clients might have low-bandwidth links, (iii) work with a dynamic client population at each round of communication between the server and the clients, as a small fraction of clients are sampled at each round. To address these challenges we develop (optimal) communication-efficient schemes for private mean estimation for several ℓ p spaces, enabling efficient gradient aggregation for each iteration of the optimization solution of the ERM. We also provide lower and upper bounds for mean estimation with privacy and communication constraints for arbitrary ℓ p spaces. To get the overall communication, privacy, and optimization performance operation point, we combine this with privacy amplification opportunities inherent to this setup. Our solution takes advantage of the inherent privacy amplification provided by client sampling and data sampling at each client (through Stochastic Gradient Descent) as well as the recently developed privacy framework using anonymization, which effectively presents to the server responses that are randomly shuffled with respect to the clients. Putting these together, we demonstrate that one can get the same privacy, optimization-performance operating point developed in recent methods that use full-precision communication, but at a much lower communication cost, i.e., effectively getting communication efficiency for “free”.
16 April 2024 in IEEE Journal on Selected Areas in Information Theory
In this paper, we propose differentially private algorithms for the problem of stochastic linear bandits in the central, local and shuffled models. In the central model, we achieve almost the same regret as the optimal non-private algorithms, which means we get privacy for free. In particular, we achieve a regret of $\tilde {O}\left({\sqrt {T}+{}\frac {1}{\varepsilon }}\right)$ matching the known lower bound for private linear bandits, while the best previously known algorithm achieves $\tilde {O}\left({{}\frac {1}{\varepsilon }\sqrt {T}}\right)$ . In the local case, we achieve a regret of $\tilde {O}\left({{}\frac {1}{\varepsilon }{\sqrt {T}}}\right)$ which matches the non-private regret for constant $\varepsilon $ , but suffers a regret penalty when $\varepsilon $ is small. In the shuffled model, we also achieve regret of $\tilde {O}\left({\sqrt {T}+{}\frac {1}{\varepsilon }}\right)$ while the best previously known algorithm suffers a regret of $\tilde {O}\left({{}\frac {1}{\varepsilon }{T^{3/5}}}\right)$ . Our numerical evaluation validates our theoretical results. Our results generalize for contextual linear bandits with known context distributions.
15 February 2024 in IEEE Journal on Selected Areas in Information Theory
We study the distributed mean estimation (DME) problem under privacy and communication constraints in the local differential privacy (LDP) and multi-message shuffled (MMS) privacy frameworks. The DME has wide applications in both federated learning and analytics. We propose a communicationefficient and differentially private algorithm for DME of bounded ℓ2-norm and ℓ∞-norm vectors. We analyze our proposed DME schemes showing that our algorithms have orderoptimal privacy-communication-performance trade-offs. Our algorithms are designed by giving unequal privacy assignments at different resolutions of the vector (through binary expansion) and appropriately combining it with coordinate sampling. These results are directly applied to give guarantees on private federated learning algorithms. We also numerically evaluate the performance of our private DME algorithms.
6 February 2023 in IEEE Transactions on Information Theory
This paper studies the Multi-Input-Multi-Output (MIMO) interference networks with arbitrary number of transmitters and receivers, where both the transmitters and receivers are equipped with caches. The main goal is to design content placement and delivery schemes that minimize the worst case normalized delivery time (NDT). First, we propose a delivery scheme for the cache-aided Single-Input-Multiple-Output (SIMO) interference networks. Then, we obtain the achievable NDT of the cache-aided MIMO interference networks using the decomposition property of splitting each multi-antenna transmitter into multiple single antenna transmitters. Furthermore, we derive an information-theoretic bound on the optimal NDT of the cache-aided MIMO interference network. Analytical results show that the proposed scheme is within a multiplicative gap of 2 from the derived lower bound independent of all system parameters for any uncoded cache placement scheme. We also derive a novel delivery scheme for the cache-aided Multi-Input-Single-Output (MISO) interference network outperforming our proposed scheme for the cache-aided MIMO interference network. The numerical results show the superiority of our proposed scheme over the state-of-the-art schemes in the literature. Our results show that the coded caching gain has a more significant contribution in reducing the transmission latency than the spatial multiplexing gain. Our results indicate that the receive-antennas become more effective in reducing the NDT than the transmit-antennas in the presence of caches at the receiver-side. In addition, we show that increasing the number of transmit-antennas has a higher gain in reducing the NDT than adding more transmitters in the cache-aided MISO interference network.
3 June 2019 in IEEE Transactions on Wireless Communications
We consider a fog radio access network (F-RAN) with multiple transmitters and receivers, where each transmitter is connected to the cloud via a fronthaul link. Each network node has a finite cache, where it fills its cache with portions of the library files in the off-peak hours. In the delivery phase, receivers request each library files according to an arbitrary popularity distribution. The cloud and the transmitters are responsible for satisfying the requests. This paper aims to design content placement and coded delivery schemes for minimizing both the expected normalized delivery time (NDT) and the peak NDT which measures the transmission latency. We propose achievable transmission policies, and derive an information-theoretic bound on the expected NDT under uniform popularity distribution. The analytical results show that the proposed scheme is within a gap of 2.58 from the derived bound for both the expected NDT under uniform popularity distribution and the peak NDT. Next, we investigate the expected NDT under an arbitrary popularity distribution for an F-RAN with transmitter-side caches only. The achievable and information-theoretic bounds on the expected NDT are derived, where we analytically prove that our proposed scheme is optimal within a gap of two independent of the popularity distribution.
25 November 2020 in IEEE Journal on Selected Areas in Information Theory
This work examines a novel question: how much randomness is needed to achieve local differential privacy (LDP)? A motivating scenario is providing multiple levels of privacy to multiple analysts, either for distribution or for heavy hitter estimation, using the same (randomized) output. We call this setting successive refinement of privacy, as it provides hierarchical access to the raw data with different privacy levels. For example, the same randomized output could enable one analyst to reconstruct the input, while another can only estimate the distribution subject to LDP requirements. This extends the classical Shannon (wiretap) security setting to local differential privacy. We provide (order-wise) tight characterizations of privacy-utility-randomness trade-offs in several cases for distribution estimation, including the standard LDP setting under a randomness constraint. We also provide a non-trivial privacy mechanism for multi-level privacy. Furthermore, we show that we cannot reuse random keys over time while preserving privacy of each user.