1 January 2024 in IEEE Signal Processing Magazine
Recently, bilevel optimization (BLO) has taken center stage in some very exciting developments in the area of signal processing (SP) and machine learning (ML). Roughly speaking, BLO is a classical optimization problem that involves two levels of hierarchy (i.e., upper and lower levels), wherein obtaining the solution to the upper-level problem requires solving the lower-level one. BLO has become popular largely because it is powerful in modeling problems in SP and ML, among others, that involve optimizing nested objective functions. Prominent applications of BLO range from resource allocation for wireless systems to adversarial ML. In this work, we focus on a class of tractable BLO problems that often appear in SP and ML applications. We provide an overview of some basic concepts of this class of BLO problems, such as their optimality conditions, standard algorithms (including their optimization principles and practical implementations) as well as how they can be leveraged to obtain state-of-the-art results for several key SP and ML applications. Further, we discuss some recent advances in BLO theory and its implications for applications, and we point out some limitations of the state of the art that require significant future research efforts. We hope that this article, together with the associated open source BLO toolbox we developed for algorithm benchmarking, can serve to accelerate the adoption of BLO as a generic tool to model, analyze, and innovate on a wide array of emerging SP and ML applications.
24 September 2021 in IEEE Transactions on Signal Processing
In this work, we propose a joint collaboration-compression framework for sequential estimation of a random vector parameter in a resource constrained wireless sensor network (WSN). Specifically, we propose a framework where the local sensors first collaborate (via a collaboration matrix) with each other. Then a subset of sensors selected to communicate with the FC linearly compress their observations before transmission. We design near-optimal collaboration and linear compression strategies under power constraints via alternating minimization of the sequential minimum mean square error. The objective function for collaboration design is generally non-convex. We establish correspondence between the sparse collaboration matrix and the non-sparse vector consisting of the nonzero elements of the collaboration matrix. Then, we reformulate and solve the collaboration design problem using quadratically constrained quadratic program (QCQP). The compression design problem is solved using the same methodology. We propose two versions of compression design, one centralized scheme where the compression strategies are derived at the FC and decentralized, where the local sensors compute their individual compression strategies independently. Importantly, we show that the proposed methods can also be used for estimating time-varying random vector parameters. Finally, numerical results are provided to demonstrate the effectiveness of the proposed framework.
19 November 2021 in IEEE Transactions on Signal and Information Processing over Networks
Distributed learning is an important paradigm in the current machine learning algorithms with large datasets. In this paper, distributed stochastic optimization problem of minimizing a nonconvex function in an adversarial setting is considered. A robust variant of the stochastic variance-reduced algorithm is proposed. In the distributed setup, we assume that a fraction of worker nodes (WNs) can be Byzantines. We assume that the batch gradients are computed at the WNs and the stochastic gradients are computed at the central node (CN). We provide the convergence rate of the proposed algorithm which employs the design of a novel filtering rule that is independent of the problem dimension. Furthermore, we capture the effect of Byzantines present in the network on the convergence performance of the algorithm. We evaluate the performance of the proposed algorithm and present the simulation results using real world datasets, in addition to providing the theoretical guarantees.
5 August 2021 in IEEE Signal Processing Letters
In this work, we propose a joint collaboration-compression framework for the random signal detection problem in a resource constrained wireless sensor network (WSN). Specifically, we propose a framework where the local sensors first collaborate (via a linear collaboration matrix) with each other. Then a subset of sensors linearly compress their aggregated information before communicating with the fusion center (FC). We propose a novel metric called generalized deflection coefficient (GDC) for evaluating the detection performance which is shown to be tightly upper bounded by the Kullback-Leibler divergence for Gaussian observations. We jointly design the linear collaboration and compression strategies under power constraints via alternating maximization of the proposed GDC metric. Finally, numerical results are provided to demonstrate the effectiveness of the proposed framework.