20 September 2023 in IEEE Transactions on Aerospace and Electronic Systems
Real-time emergency event processing for mega-constellation management requires an approach that can quickly obtain a satisfactory satellite range rescheduling scheme. In this paper, a deep reinforcement learning (DRL) approach is proposed for rapid rescheduling on the basis of established satellite range scheduling schemes. Three rescheduling principles with different impacts on established scheduling schemes are first analyzed. A mathematical model for both maximizing the completion profits of dynamic missions and minimizing their impacts on scheduled missions is formulated. An efficient DRL-based approach to satellite range rescheduling is then developed. A mask processing mechanism is integrated into the reinforcement learning framework to effectively eliminate the conflicts accurately expressed by the conflict space, and a modified pointer network (Ptr-Net) based on the mask processing mechanism is employed to approximate the action-value function for optimal decision-making. Experimental results show that the proposed approach significantly outperforms the rule-based heuristic and another DRL-based method in terms of computational efficiency and optimization performance, and the total time spent during the training and testing phases of the proposed approach is approximately 13% of the total running time required by the search-based methods.
1 January 2021 in Computational Intelligence and Neuroscience
Reinforcement learning from demonstration (RLfD) is considered to be a promising approach to improve reinforcement learning (RL) by leveraging expert demonstrations as the additional decision‐making guidance. However, most existing RLfD methods only regard demonstrations as low‐level knowledge instances under a certain task. Demonstrations are generally used to either provide additional rewards or pretrain the neural network‐based RL policy in a supervised manner, usually resulting in poor generalization capability and weak robustness performance. Considering that human knowledge is not only interpretable but also suitable for generalization, we propose to exploit the potential of demonstrations by extracting knowledge from them via Bayesian networks and develop a novel RLfD method called Reinforcement Learning from demonstration via Bayesian Network‐based Knowledge (RLBNK). The proposed RLBNK method takes advantage of node influence with the Wasserstein distance metric (NIW) algorithm to obtain abstract concepts from demonstrations and then a Bayesian network conducts knowledge learning and inference based on the abstract data set, which will yield the coarse policy with corresponding confidence. Once the coarse policy’s confidence is low, another RL‐based refine module will further optimize and fine‐tune the policy to form a (near) optimal hybrid policy. Experimental results show that the proposed RLBNK method improves the learning efficiency of corresponding baseline RL algorithms under both normal and sparse reward settings. Furthermore, we demonstrate that our RLBNK method delivers better generalization capability and robustness than baseline methods.
16 March 2023 in CAAI Transactions on Intelligence Technology
Policy evaluation (PE) is a critical sub‐problem in reinforcement learning, which estimates the value function for a given policy and can be used for policy improvement. However, there still exist some limitations in current PE methods, such as low sample efficiency and local convergence, especially on complex tasks. In this study, a novel PE algorithm called Least‐Squares Truncated Temporal‐Difference learning (LST 2 D) is proposed. In LST 2 D, an adaptive truncation mechanism is designed, which effectively takes advantage of the fast convergence property of Least‐Squares Temporal Difference learning and the asymptotic convergence property of Temporal Difference learning (TD). Then, two feature pre‐training methods are utilised to improve the approximation ability of LST 2 D. Furthermore, an Actor‐Critic algorithm based on LST 2 D and pre‐trained feature representations (ACLPF) is proposed, where LST 2 D is integrated into the critic network to improve learning‐prediction efficiency. Comprehensive simulation studies were conducted on four robotic tasks, and the corresponding results illustrate the effectiveness of LST 2 D. The proposed ACLPF algorithm outperformed DQN, ACER and PPO in terms of sample efficiency and stability, which demonstrated that LST 2 D can be applied to online learning control problems by incorporating it into the actor‐critic architecture.