2 November 2020 in IEEE Transactions on Neural Networks and Learning Systems
Deep reinforcement learning (DRL) has been shown to be successful in many application domains. Combining recurrent neural networks (RNNs) and DRL further enables DRL to be applicable in non-Markovian environments by capturing temporal information. However, training of both DRL and RNNs is known to be challenging requiring a large amount of training data to achieve convergence. In many targeted applications, such as those used in the fifth-generation (5G) cellular communication, the environment is highly dynamic, while the available training data is very limited. Therefore, it is extremely important to develop DRL strategies that are capable of capturing the temporal correlation of the dynamic environment requiring limited training overhead. In this article, we introduce the deep echo state Q-network (DEQN) that can adapt to the highly dynamic environment in a short period of time with limited training data. We evaluate the performance of the introduced DEQN method under the dynamic spectrum sharing (DSS) scenario, which is a promising technology in 5G and future 6G networks to increase the spectrum utilization. Compared with conventional spectrum management policy that grants a fixed spectrum band to a single system for exclusive access, DSS allows the secondary system to share the spectrum with the primary system. Our work sheds light on the application of an efficient DRL framework in highly dynamic environments with limited available training data.
19 March 2024 in IEEE Transactions on Wireless Communications
Due to its ubiquitous and contact-free nature, the use of WiFi infrastructure for performing sensing tasks has tremendous potential. However, the channel state information (CSI) measured by a WiFi receiver suffers from errors in both its gain and phase, which can significantly hinder sensing tasks. By analyzing these errors from different WiFi receivers, a mathematical model for these gain and phase errors is developed in this work. Based on these models, several theoretically justified preprocessing algorithms for correcting such errors at a receiver and, thus, obtaining clean CSI are presented. Simulation results show that at typical system parameters, the developed algorithms for cleaning CSI can reduce noise by 40% and 200%, respectively, compared to baseline methods for gain correction and phase correction, without significantly impacting computational cost. The superiority of the proposed methods is also validated in a real-world test bed for respiration rate monitoring (an example sensing task), where they improve the estimation signal-to-noise ratio by 20% compared to baseline methods.
10 January 2023 in IEEE Transactions on Wireless Communications
Dynamic spectrum access (DSA) has been introduced as a promising technology that allows a secondary system to access the licensed spectrum of the primary system to improve spectrum utilization. In this paper, we introduce Fed-MADRL by incorporating federated learning (FL) and multi-agent deep reinforcement learning (MADRL) to design a collaborative DSA strategy. Our Fed-MADRL scheme employs FL to enable multiple users to collaboratively optimize the system goal without sharing their training data. By keeping all the training data at the user end, FL improves the communication efficiency and strengthens user data privacy. To further reduce the communication overheads, each user only shares quantized information. We provide the convergence analysis to characterize the trade-off between the communication efficiency and the system performance. In particular, we show that the introduced method converges at a rate O (1/ K 1/4 ), where K is the number of FL iterations. To the best of our knowledge, Fed-MADRL is the first work that utilizes FL in DSA networks under quantized communication. Performance evaluation results show that the introduced Fed-MADRL method outperforms the independent learning method and achieves comparable performance with the centralized MADRL method, which requires much higher communication overheads.
31 August 2022 in IEEE/ACM Transactions on Networking
Mobility load balancing (MLB) aims to solve the problem of uneven resource utilization in cellular networks. Since network dynamics are usually complicated and non-stationary, conventional model-based MLB methods fail to cover all scenarios of cellular networks. On the other hand, deep reinforcement learning (DRL) can provide a flexible framework to learn to distribute cell load evenly without explicit modeling of the underlying network dynamics. In this paper, we introduce a novel decentralized DRL-based MLB method where each cell has a DRL agent to learn its handover parameters and antenna tilt angle. As the number of cells increases, the decentralized framework is more computationally efficient than its centralized counterpart by dividing the action space. Furthermore, our designed decentralized DRL architecture only requires readily known information defined in existing cellular standards, and it can achieve a more balanced cell load distribution than the centralized DRL one by using individual reward functions. To provide realistic performance evaluation, a network simulator is introduced strictly following the Third Generation Partnership Project (3GPP) specifications. Furthermore, field data is used to construct the underlying cellular environment. Extensive evaluations have been conducted to demonstrate the fact that the introduced decentralized DRL-based MLB method can achieve a more balanced cell load distribution and a better performance of edge users than the state-of-the-art MLB methods.
5 August 2024 in IEEE Transactions on Wireless Communications
In dynamic spectrum access (DSA) networks, secondary users (SUs) need to opportunistically access primary users’ (PUs) radio spectrum without causing significant interference. Since the SU-PU interaction is limited, deep reinforcement learning has been introduced to help SUs conduct spectrum access. Specifically, deep recurrent Q network (DRQN) has been utilized in DSA networks for SUs to aggregate information from recent experiences to make spectrum access decisions. DRQN is notorious for its sample efficiency since it needs a rather large number of training samples to tune its parameters which is a computationally demanding task. Deep echo state network (DEQN) has been introduced for DSA networks to address the sample efficiency issue of DRQN. In this work, we compare the convergence of DRQN and DEQN by comparing the upper bounds we obtain on their covering number, a notion of richness. Furthermore, we introduce a method to determine the right hyper-parameters for DEQN, providing system design guidance for DEQN-based DSA networks. Extensive performance evaluation confirms that DEQN-based DSA strategy is the superior choice with regard to computational power while outperforming DRQN-based ones.
9 December 2024 in IEEE Transactions on Wireless Communications
This paper focuses on advancing reinforcement learning for challenging environments characterized by partial observability and non-stationarity, such as dynamic spectrum access (DSA). In the literature, the Deep Recurrent Q-Network was introduced to capitalize on the inherent temporal correlations present in DSA. Nevertheless, its practicality is still questionable due to sample inefficiency and slow convergence. We introduce Dyna-ESN, leveraging both model-based and model-free methods by employing Reservoir Computing for generative modeling. Specifically, we utilize Echo State Networks (ESNs) to synthesize samples for enhancing the sample efficiency of a model-free Deep Echo State Q-network, enabling effective operation of agent given limited genuine relevant samples obtained through interaction with environment. To mitigate potential adverse effects of synthetic samples, an evaluation algorithm guides the sample selection process, ensuring reliability. A sample augmentation technique is also introduced to allow agents to collect adequate samples despite controlling the sensing rate and duration of secondary transmissions. Our analysis explores trade-offs between data evaluation and sample efficiency, as well as the bias-variance trade-off of the model, identifying optimal design parameters. Evaluating the performance of Dyna-ESN in DSA scenarios demonstrates its performance benefits over existing methods, paving the way for more efficient and effective techniques in complex dynamic environments.