In this paper, we study the handover (HO) and power allocation problem in a two-tier heterogeneous network (HetNet), which consists of a macro base station and some millimeter-wave (mmWave) small base stations. We establish an HO management and power allocation scheme to maximize the overall throughput while reducing the HO frequency. In particular, considering the interrelationship among decisions made by different user equipments (UEs), we first model the HO and power allocation problem as a fully cooperative multi-agent task, in which all agents, i.e., UEs, have the same target. Then, to solve the multi-agent task and get decentralized policies for each UE, we develop a multi-agent reinforcement learning (MARL) algorithm based on the proximal policy optimization (PPO) method, by introducing the centralized training with decentralized execution framework. That is, we use global information to train policies for each UE, and after training is finished, each UE obtains a decentralized policy, which can be implemented only based on each UE's local observation. Specially, we introduce the counterfactual baseline to address the credit assignment problem in centralized learning. Due to the centralized training, the decentralized polices learned by multi-agent PPO (MAPPO) can work more cooperatively. Finally, the simulation results demonstrate that our method can achieve better performance comparing with other existing works.

This paper considers an adversarial scenario between a legitimate eavesdropper and a suspicious communication pair. All three nodes are equipped with multiple antennas. The eavesdropper, which operates in a full-duplex model, aims to wiretap the dubious communication pair via proactive jamming. On the other hand, the suspicious transmitter, which can send artificial noise (AN) to disturb the wiretap channel, aims to guarantee secrecy. More specifically, the eavesdropper adjusts jamming power to enhance the wiretap rate, while the suspicious transmitter jointly adapts the transmit power and noise power against the eavesdropping. Considering the partial observation and complicated interactions between the eavesdropper and the suspicious pair in unknown system dynamics, we model the problem as an imperfect-information stochastic game. To approach the Nash equilibrium solution of the eavesdropping game, we develop a multi-agent reinforcement learning (MARL) algorithm, termed neural fictitious self-play with soft actor-critic (NFSP-SAC), by combining the fictitious self-play (FSP) with a deep reinforcement learning algorithm, SAC. The introduction of SAC enables FSP to handle the problems with continuous and high dimension observation and action space. The simulation results demonstrate that the power allocation policies learned by our method empirically converge to a Nash equilibrium, while the compared reinforcement learning algorithms suffer from severe fluctuations during the learning process.

In this paper, we investigate the transmission schemes of space data systems for optimized network capacity in an integrated terrestrial-satellite network (ITSN) with a twolayered space segment. First, a theoretical model of the network capacity is developed to evaluate the strategy of utilizing both direct and relayed transmissions. Second, we consider the ideal and the resource-constrained scenarios in which the corresponding network capacity is modeled with respect to the scheduling scheme. In particular, closed form and semi-closed form solutions to the difficult integer programs are achieved via rigorous mathematical analysis. The proposed model is general for exploring the capacity of various satellite network deployments whose solutions have not been obtained in prior studies. Furthermore, we verify the potential capacity of the different transmission schemes based on the proposed solutions and prove that the system's network capacity can be significantly improved by the hybrid transmission scheme. The theoretical framework proposed in this paper is expected to provide constructive insights in the design for the future space segments of ITSN.