11 June 2024 in IEEE Transactions on Vehicular Technology
The next-generation wireless network is expected to use low- earth orbit (LEO) satellite networks to deliver seamless and high-capacity global communications services. Due to the high-speed mobility of LEO satellites, massive and frequent handovers inevitably occur. Moreover, handover becomes more complicated with the ever-growing constellation scale, number of mobile terminals (MTs), and demands for emerging delay-sensitive applications. In this paper, a decentralized Markov decision process (DEC-MDP) is adopted to formulate the handover problem in the LEO satellite network with finite bursty traffic. The target is maximizing the total reward associated with the service revenue and the cost of handover and packet loss. To deal with the high computational complexity caused by the large state space and action space, the solution is designed using a multi-agent double deep Q-network (MADDQN) with fully decentralized framework, which also allows each MT to train and use an individual local DDQN to avoid load imbalance between satellites. Further, to alleviate the non-stationary issue of the environment in parallel learning, multi-agent fingerprints are applied in MADDQN, and the proposed algorithm is called multi-agent fingerprints-enhanced double deep Q-network-based distributed intelligent handover (MAF-DDQN-DIH) mechanism. The implementation of MAF-DDQN-DIH in practical communication systems are discussed, and the corresponding communication overhead and computational complexity are analyzed. Simulation results demonstrate that the designed multi-agent fingerprints are effective and the proposed MAF-DDQN-DIH algorithm outperforms the comparison handover algorithms in terms of the total reward.
21 July 2022 in IEEE Transactions on Wireless Communications
Heterogeneous networks (HetNets) with end-to-end (E2E) network slicing are regarded as effective approaches to meet diverse service requirements from vertical industries. Due to the dense deployment of base stations (BSs) and the complicated associations between BSs and E2E network slices (NSs) in the scenario, the handoff problem faces challenges of the huge system state space and handoff action space and the considerable communication overhead. In this paper, we take these issues into account and consider a distributed E2E NS handoff decision framework in the HetNet. A decentralized Markov decision process (DEC-MDP)-based model is formulated for the distributed E2E NS handoff problem, and the jointly observable and random characteristics of the DEC-MDP are analyzed. To obtain a theoretical performance reference, the original distributed E2E NS handoff problem is simplified, and a Nash equilibrium-based performance bound is given. More practically, the multi-agent double deep Q-network-based distributed handoff (MA-DDQN-DH) algorithm with the centralized training and decentralized executing framework is proposed. Simulation results show that the Nash equilibrium-based performance bound is reasonable, and the proposed MA-DDQN-DH algorithm performs well in the comparison.
23 June 2025 in IEEE Transactions on Vehicular Technology
End-to-end network slicing in unmanned aerial vehicle (UAV) networks has attracted significant attention for delivering differentiated services tailored to the quality of service (QoS) requirements of user terminals (UTs). In a sliced UAV network, the dual mobility of both UTs and UAVs, combined with the three-layer association of UT-UAV-network slice (NS), makes handoff a critical and challenging problem for maintaining the communication performance of UTs. This paper proposes a sequential handoff process where a central controller makes handoff decisions for each UT sequentially according to the UT queue. This process is modeled as a Markov decision process (MDP). In the MDP, the state space includes NS available resources, UTs' QoS requirements, and UT queue information. Additionally, the reward function accounts for UTs' QoS and handoff overhead. The double deep Q-network (DDQN) technique is then employed to design a handoff algorithm. This algorithm achieves the optimization goal of ensuring UTs' QoS requirements while avoiding frequent handoffs by maximizing the total reward of the MDP. Additionally, a state-dependent exploration technique with adaptive intensity adjustment, called noisy network (NoisyNet), is utilized to drive the DDQN for effective exploration, thereby enhancing the performance of the handoff algorithm. Finally, the runtime complexity of the proposed NoisyNet-DDQN-based sequential handoff (NoisyNet-DDQN-SH) algorithm is analyzed. The simulation results show that the NoisyNet-DDQN-SH algorithm converges well. Moreover, it exhibits strong robustness. Additionally, the NoisyNet technique effectively enhances the algorithm's performance, making the NoisyNet-DDQN-SH algorithm superior to the vanilla DDQN (with an ϵ-greedy strategy), greedy, maximum signal-to-interference-plus-noise ratio (MaxSINR), and random algorithms.
23 June 2025 in IEEE Internet of Things Journal
Due to the triple mobility of mobile users (MUs), unmanned aerial vehicle (UAV) relays, and low Earth orbit (LEO) satellites, handover becomes a critical and challenging issue for maintaining the continuity and quality of communication services in UAV-assisted LEO satellite networks. This article proposes a distributed handover decision-making process aimed at improving scalability and reducing communication overhead. The handover problem is modeled as a decentralized Markov decision process (DEC-MDP) with the objective of maximizing the total end-to-end (E2E) throughput. We design an independent proximal policy optimization-based distributed intelligent handover (IPPO-DIH) algorithm within a centralized training with decentralized execution framework to solve the DEC-MDP. To analyze the theoretical optimal E2E throughput, we eliminate the correlation between handover decisions at different time steps. A three-sided matching algorithm with theoretical convergence guarantees is designed to obtain a stable matching among MUs, UAV relays, and LEO satellites at each time step. These stable matchings are combined to provide a theoretical performance benchmark for the handover algorithms. Simulation results validate the convergence of the proposed IPPO-DIH and three-sided matching algorithms. Additionally, the total E2E throughput achieved by the IPPO-DIH algorithm approaches the theoretical performance benchmark and outperforms typical handover algorithms.
1 January 2026 in IEEE Transactions on Wireless Communications