Invalid action masking is a practical technique in deep reinforcement learning to prevent agents from taking invalid actions. Existing approaches rely on action masking during policy training and utilization. This study focuses on developing reinforcement learning algorithms that incorporate action masking during training but can be used without action masking during policy execution. The study begins by conducting a theoretical analysis to elucidate the distinction between naive policy gradient and invalid action policy gradient. Based on this analysis, we demonstrate that the naive policy gradient is a valid gradient and is equivalent to the proposed composite objective algorithm, which optimizes both the masked policy and the original policy in parallel. Moreover, we propose an off-policy algorithm for invalid action masking that employs the masked policy for sampling while optimizing the original policy. To compare the effectiveness of these algorithms, experiments are conducted using a simplified real-time strategy (RTS) game simulator called Gym-$\mu$RTS. Based on empirical findings, we recommend utilizing the off-policy algorithm for addressing most tasks while employing the composite objective algorithm for handling more complex tasks.

Unmanned aerial vehicles (UAVs) have numerous applications in reconnaissance, search, and rescue missions. However, searching underwater areas remains a challenge for UAVs. For anti-submarine search tasks, this paper proposes using a mixed swarm of UAVs and unmanned surface vehicles (USVs). To generate vehicle paths with the goal of minimizing the travel time while achieving complete coverage, a two-phase strategy is developed. In the first phase, optimal waypoints are determined to balance the task volume among the UAVs and USVs. A mapping model is constructed to represent the relationships between the different unmanned platforms and the mission environment. In the second phase, the search trajectory for each UAV/USV is generated by combining waypoint clustering and closed-loop trajectory generation. A detailed simulation study confirms the feasibility and flexibility of the proposed path planning method when applied to cross-domain swarms of different types and sizes for anti-submarine missions. In addition, the simulation results show that cross-domain cooperative planning for unmanned platforms may have advantages over single-domain planning in some scenarios.

Autonomous decision-making for air confrontation between unmanned combat aerial vehicles remains hard to be designed due to dynamic situations and complex interactions. Rule-based decision-making methods provide a powerful solution with better interpretability. However, various hand-crafted rules may result in conflicts and poor scalability issues. To overcome this problem, this work proposes a hierarchical decision-making framework called State-Event-Condition-Action (SECA), which integrates the finite state machine and event-condition-action frameworks. This framework provides three products for system design: the SECA model–an abstract model of rules; the SECA state chart–a graphical visualization of rules; and the SECA rule description–a machine-readable format for practical deployment. The SECA framework offers several advantages, including convenient deployment, high efficiency, better logicality, and scalability. Simulation results demonstrate that the SECA framework enables autonomous decision-making in air confrontation scenarios and outperforms the event-condition-action framework in terms of computational time and cost-effectiveness. Furthermore, the generalization test in robot navigation tasks verifies its potential applicability to other domains with different background knowledge.