Beam-hopping scheduling technology is a crucial technology for improving the performance of resource-constrained multibeam satellite (MBS) systems. However, it is challenging to dynamically match the communication demands of the terrestrial cells with the beam transmission capacity. Almost all state-of-art beam-hopping scheduling approaches only consider a fixed beam coverage radius, which always results in a waste of transmission resources. In this letter, we propose a deep reinforcement learning-based algorithm to jointly optimize the beam-hopping scheduling and coverage control (called DeepBeam) which flexibly uses three degrees of freedom of time, space and beam coverage radius. In addition, to reduce the computational complexity and accelerate the training of DeepBeam, we first train a single model for one of the beams and then use this trained model for each beam in a polling manner to determine the cell to be illuminated and the beam coverage. Furthermore, the simulation results illustrate that the proposed DeepBeam can improve not only the throughput of MBS but also reduce the packet loss probability.

Due to the explosive growth in the scale of low Earth orbit (LEO) satellite networks, such as Starlink, and the extremely high mobility and limited network resources of single LEO satellites, designing dynamic routing has become a promising way to improving satellite communication performance. Most of the existing approaches derive the routing policies in a centralized paradigm, which often suffer from the high cost of collecting the global routing information and the high computation complexity in large-scale networks. Therefore, this letter proposes a spatial location aided fully-distributed routing algorithm for large-scale satellite network with minimizing the average delivery time. First, we model a partially observable Markov decision process (POMDP) for each satellite, where only the spatial location, queuing state and estimated residual propagation delay (ERPD) of one-hop adjacent satellites need to be communicated. Second, a novel Fully Distributed dynamic Routing algorithm based on Multi-Agent deep Reinforcement Learning (FDR-MARL) is proposed to derive the optimal routing strategy. Finally, extensive experiments are carried out to verify the effectiveness and advantages of our proposed approach under large-scale LEO satellite networks.

The rapid advancements in cloud computing, Big Data and their related applications have led to a skyrocketing increase in data center energy consumption year by year. The prior approaches for improving data center energy efficiency mostly suffer from high system dynamics or the complexity of data centers. In this paper, we propose an optimization framework based on deep reinforcement learning, named DeepEE, to jointly optimize energy consumption from the perspectives of task scheduling and cooling control. In DeepEE, a PArameterized action space based Deep Q-Network (PADQN) algorithm is proposed to tackle the hybrid action space problem. Then, a dynamic time factor mechanism for adjusting cooling control interval is introduced into PADQN (PADQN-D) to achieve more accurate and efficient coordination of IT and cooling subsystems. Finally, in order to train and evaluate the proposed algorithms safely and quickly, a simulation platform is built to model the dynamics of IT and cooling subsystems. Extensive real-trace based experiments illustrate that: 1) the proposed PADQN algorithm can save up to 15% and 10% energy consumption compared with the baseline siloed and joint optimization approaches respectively; 2) the proposed PADQN-D algorithm with dynamic cooling control interval can better adapt to the change of IT workload; 3) our proposed algorithms achieve more stable performance gain in terms of power consumption by adopting the parameterized action space.

Low-Earth orbit (LEO) satellites utilizing beam hopping (BH) technology offer extensive coverage, low latency, high bandwidth, and significant flexibility. However, the uneven geographical distribution and temporal variability of ground traffic demands, combined with the high mobility of LEO satellites, present significant challenges for efficient beam resource utilization. Traditional BH methods based on GEO satellites fail to address issues such as satellite interference, overlapping coverage, and mobility. This paper explores a Digital Twin (DT)-based collaborative resource allocation network for multiple LEO satellites with overlapping coverage areas. A two-tier optimization problem, focusing on load balancing and cell service fairness, is proposed to maximize throughput and minimize inter-cell service delay. The DT layer optimizes the allocation of overlapping coverage cells by designing BH patterns for each satellite, while the LEO layer optimizes power allocation for each selected service cell. At the DT layer, an Actor-Critic network is deployed on each agent, with a global critic network in the cloud center. The A3C algorithm is employed to optimize the DT layer. Concurrently, the LEO layer optimization is performed using a Multi-Agent Reinforcement Learning algorithm, where each beam functions as an independent agent. The simulation results show that this method reduces satellite load disparity by about 72.5% and decreases the average delay to 12ms. Additionally, our approach outperforms other benchmarks in terms of throughput, ensuring a better alignment between offered and requested data.

To reduce the skyrocketing energy consumption of data centers, the prevailing approaches adopt the time-driven manner to control IT and cooling subsystems. These methods suffer from highly dynamic system states, complex action spaces and the risk of instability caused by frequent and unnecessary control operations. To tackle these problems, we propose a novel event-driven control paradigm and an optimization algorithm, under the deep reinforcement learning (DRL) framework. The principle is to make decisions based on certain critical events (e.g., overheating), rather than fixed periodic control. Specifically, we design an event-driven optimization framework to trigger control operations. Then, we present several models to describe IT and cooling subsystems, and mathematically define events to capture four types of prior factors that impact system performance. Furthermore, we develop an event-driven DRL (E-DRL) optimization algorithm to dispatch jobs and regulate cooling facilities for energy efficiency. Using two different types of real workload traces, we conduct extensive experiments to demonstrate that: 1) E-DRL reduces the number of regulating decisions by 70%-95% while achieving a comparable or even better energy efficiency in comparison with the state-of-the-art algorithm; and 2) E-DRL can adapt the control frequency to the changing operational conditions and diverse workloads.

An efficient routing strategy in Low Earth Orbit (LEO) satellite networks is critical for air-space-ground integrated communication towards 6G. However, the existing terrestrial routing algorithms cannot well handle the high-speed movement issue of satellites, while the existing satellite routing algorithms usually suffer from high communication overhead or high computational complexity. To address the above issues, we propose a fully distributed dynamic packet routing algorithm based on Graph Neural Network (GNN)-enhanced Multi-Agent Deep Reinforcement Learning (MADRL), named GraphPR. In GraphPR, the satellite routing problem is modeled as a Partially Observable Markov Decision Process (POMDP), where each satellite only needs to share the information with one-hop neighbors. Then, Graph Attention Network (GAT) is leveraged to encode the perceived one-hop information and derive a hidden representation implicitly consisting of multi-hop satellite information. Subsequently, MADRL is employed to build a fully distributed optimization framework and make routing decisions. In addition, a special mechanism called Residual Shortest Path Hops (RSPH) is designed to guide the routing selection and avoid routing loops. Finally, the experimental results illustrate that GraphPR has better performance in terms of packet loss rate, average delivery time, throughput, and average queuing length than the baseline algorithms.