Invalid action masking is a practical technique in deep reinforcement learning to prevent agents from taking invalid actions. Existing approaches rely on action masking during policy training and utilization. This study focuses on developing reinforcement learning algorithms that incorporate action masking during training but can be used without action masking during policy execution. The study begins by conducting a theoretical analysis to elucidate the distinction between naive policy gradient and invalid action policy gradient. Based on this analysis, we demonstrate that the naive policy gradient is a valid gradient and is equivalent to the proposed composite objective algorithm, which optimizes both the masked policy and the original policy in parallel. Moreover, we propose an off-policy algorithm for invalid action masking that employs the masked policy for sampling while optimizing the original policy. To compare the effectiveness of these algorithms, experiments are conducted using a simplified real-time strategy (RTS) game simulator called Gym-$\mu$RTS. Based on empirical findings, we recommend utilizing the off-policy algorithm for addressing most tasks while employing the composite objective algorithm for handling more complex tasks.

This paper investigates a distributed time-varying optimization problem concerning UAV swarm systems with bounded inputs. The objective is to achieve state consensus and minimize the sum of local time-varying objective functions in a distributed manner, while considering the constraints on inputs. To address this problem, we propose a distributed optimization protocol utilizing the projection operator and prediction-correction method. This protocol enables the UAVs to track the time-varying optimal solution with an asymptotically diminishing error. It is shown that the proposed protocol achieves approximate global optimal consensus for UAVs with different connected communication topologies, including undirected, directed detail-balanced, and switching topologies. Consequently, the UAV swarm, equipped with the proposed protocol, can establish a predetermined formation and effectively track moving targets. To validate the theoretical findings, we present three applications: a scalar example, a cooperative hunting task, and swarm tracking behavior.

In order to meet the higher accuracy requirement of trajectory prediction for Unmanned Aircraft System (UAS) in Unmanned Aircraft System Traffic Management (UTM), an Intent Based Trajectory Prediction and Smooth Based on Constrained State-dependent-transition Hybrid Estimation (CSDTHE-IBTPS) algorithm is proposed. Firstly, an intent inference method of UAS is constructed based on the information of ADS-B and geofence system. Moreover, a geofence layering algorithm is proposed. Secondly, the Flight Mode Change Points (FMCP) are used to define the relevant mode transition parameters and design the guard conditions, so as to generate the mode transition probability matrix and establish the continuous state-dependent-transition model. After that, the constrained Kalman filter (CKF) is applied to improve State-dependent-transition Hybrid Estimation (SDTHE) algorithm by applying equality constraint to the velocity of UAS in the straight phase and turning phase, respectively, and thus the constrained state-dependent-transition hybrid estimation (CSDTHE) algorithm is constructed. Finally, the results of intent inference and hybrid estimation are used to make trajectory prediction. Furthermore, each flight segment of trajectory is smoothed respectively by Rauch-Tung-Striebel (RTS) backward smooth method using the proposed CSDTHE-RTS algorithm, so as to obtain more accurate trajectory prediction results. The simulation shows that the proposed algorithm can reduce the errors of trajectory prediction and the time delay of intent inference.

Consensus control of multi-agent systems has attracted compelling attentions from various scientific communities for its promising applications. This paper presents a discrete-time consensus protocol for a class of multi-agent systems with switching topologies and input constraints based on distributed predictive control scheme. The consensus protocol is not only distributed but also depends on the errors of states between agent and its neighbors. We focus mainly on dealing with the input constraints and a distributed model predictive control scheme is developed to achieve stable consensus under the condition that both velocity and acceleration constraints are included simultaneously. The acceleration constraint is regarded as the changing rate of velocity based on some reasonable assumptions so as to simplify the analysis. Theoretical analysis shows that the constrained system steered by the proposed protocol achieves consensus asymptotically if the switching interaction graphs always have a spanning tree. Numerical examples are also provided to illustrate the validity of the algorithm.

Addressing prevalent challenges in current cooperative task assignment methods for cross-domain unmanned swarm, such as the disconnection between decision-making and execution processes, and the inadequate incorporation of platform kinematic constraints, this study introduces an integrated decision-control cooperative task assignment approach based on a bi-level optimization framework. The proposed framework formulates a bi-level programming model that tightly couples upper-level task assignment with lower-level optimal control. The upper-level model aims to minimize the maximum task completion time by optimizing the assignment and visitation sequences of diverse target types across heterogeneous unmanned platforms. The lower-level model, given the task sequences from the upper level, addresses a minimum-time optimal control problem based on a comprehensive nonlinear kinematic model. This approach enables precise computation of task execution times, which are subsequently fed back to the decision-making layer, thereby establishing a closed-loop optimization mechanism. To solve this complex model efficiently, the lower-level employs differential flatness transformation to eliminate trigonometric functions in the kinematic equations and discretizes the continuous-time optimal control problem into a nonlinear programming problem via the Radau pseudospectral method. For the upper-level combinatorial optimization, an improved genetic algorithm is developed, integrating hybrid encoding, dual-archive elitism preservation, adaptive crossover and mutation strategies, and periodic local search. Simulation results demonstrate that, compared with traditional Euclidean-distance-based assignment methods, the proposed approach generates kinematically feasible and smooth trajectories while thoroughly accounting for the kinematic constraints of heterogeneous platforms, thereby demonstrating its effectiveness and superiority in improving the comprehensive mission performance of cross-domain unmanned swarms.