Unmanned aerial vehicles (UAVs) are commonly employed in pursuit and rescue missions, where the target’s trajectory is unknown. Traditional methods, such as evolutionary algorithms and ant colony optimization, can generate a search route in a given scenario. However, when the scene changes, the solution needs to be recalculated. In contrast, more advanced deep reinforcement learning methods can train an agent that can be directly applied to a similar task without recalculation. Nevertheless, there are several challenges when the agent learns how to search for unknown dynamic targets. In this search task, the rewards are random and sparse, which makes learning difficult. In addition, because of the need for the agent to adapt to various scenario settings, interactions required between the agent and the environment are more comparable to typical reinforcement learning tasks. These challenges increase the difficulty of training agents. To address these issues, we propose the OC-MAPPO method, which combines optimal control (OC) and Multi-Agent Proximal Policy Optimization (MAPPO) with GPU parallelization. The optimal control model provides the agent with continuous and stable rewards. Through parallelized models, the agent can interact with the environment and collect data more rapidly. Experimental results demonstrate that the proposed method can help the agent learn faster, and the algorithm demonstrated a 26.97% increase in the success rate compared to genetic algorithms.

In this paper, we propose a three-layer distributed simulation network architecture, which consists of a distributed virtual simulation network, a perception and control subnetwork, and a cooperative communication service network. The simulation architecture runs on a distributed platform, which can provide unique virtual scenarios and multiple simulation services for the verification of basic perception, control, and planning algorithms of a single-robot system and can verify the distributed collaboration algorithms of heterogeneous multirobot systems. Further, we design simulation experimental scenarios for classic heterogeneous robotic systems such as unmanned aerial vehicles (UAVs) and unmanned ground vehicles (UGVs). Through the analysis of experimental measurement data, we draw several important conclusions: firstly, the replication time characteristics and update frequency characteristics of entity synchronization in our system indicate that the replication time of entity synchronization in our system is relatively short, and the update frequency can meet the needs of multirobot collaboration and ensure the real-time use and accuracy of the system; secondly, we analyze the bandwidth usage of data frames in the whole session and observe that the server side occupies almost half of the data throughput during the whole session, which indicates that the allocation and utilization of data transmission in our system is reasonable; and finally, we construct a bandwidth estimation surface model to estimate the bandwidth requirements of the current model when scaling the server-side scale and synchronization-state scale, which provides an important reference for better planning and optimizing of the resource allocation and performance of the system. Based on this distributed simulation framework, future research will improve the key technical details, including further refining the coupling object dynamic model update method to support the simulation theory of the coupling relationship between system objects, studying the impact of spatiotemporal consistency of distributed systems on multirobot control and decision making, and in-depth research on the impact of collaborative frameworks combined with multirobot systems for specific tasks.

This paper proposed a three-layer distributed simulation network architecture: 1. Distributed virtual simulation network, 2. Perception and control sub-network, and 3. Collaborative communication service network. This simulation architecture runs on a distributed platform. The platform can provide a unique virtual scenario and several simulation services to verify the individual robot system's basic perception, control, and planning algorithms and the distributed collaborative algorithms of heterogeneous multi-robot systems. Furthermore, we designed a simulation experiment scenario for classical heterogeneous robot systems such as unmanned aerial vehicles (UAVs) and unmanned ground vehicles (UGVs). At last, We gave a rough estimation model of the bandwidth.

In multi-agent domains, dealing with non-stationary opponents that change behaviors (policies) consistently over time is still a challenging problem, where an agent usually requires the ability to detect the opponent’s policy accurately and adopt the optimal response policy accordingly. Previous works commonly assume that the opponent’s observations and actions during online interactions are known, which can significantly limit their applications, especially in partially observable environments. This paper focuses on efficient policy detecting and reusing techniques against non-stationary opponents without their local information. We propose an algorithm called Bayesian policy reuse with LocAl oBservations (Bayes-Lab) by incorporating variational autoencoders (VAE) into the Bayesian policy reuse (BPR) framework. Following the centralized training with decentralized execution (CTDE) paradigm, we train VAE as an opponent model during the offline phase to extract the latent relationship between the agent’s local observations and the opponent’s local observations. During online execution, the trained opponent models are used to reconstruct the opponent’s local observations, which can be combined with episodic rewards to update the belief about the opponent’s policy. Finally, the agent reuses the best response policy based on the updated belief to improve online performance. We demonstrate that Bayes-Lab outperforms existing state-of-the-art methods in terms of detection accuracy, accumulative rewards, and episodic rewards in a predator–prey scenario. In this experimental environment, Bayes-Lab can achieve about 80% detection accuracy and the highest accumulative rewards, and its performance is less affected by the opponent policy switching interval. When the switching interval is less than 10, its detection accuracy is at least 10% higher than other algorithms.

Accurate and feasible target assignment in an urban environment without road networks remains challenging. Existing methods exhibit critical limitations: computational inefficiency preventing real-time decision-making requirements and poor cross-scenario generalization, yielding task-specific policies that lack adaptability. To achieve efficient target assignment in urban adversarial scenarios, we propose an efficient traversable path generation method requiring only binarized images, along with four key constraint models serving as optimization objectives. Moreover, we model this optimization problem as a Markov decision process (MDP) and introduce the generalization sequential proximal policy optimization (GSPPO) algorithm within the reinforcement learning (RL) framework. Specifically, GSPPO integrates an exploration history representation module (EHR) and a neuron-specific plasticity enhancement module (NPE). EHR incorporates exploration history into the policy learning loop, which significantly improves learning efficiency. To mitigate the plasticity loss in neural networks, we propose an NPE module, which boosts the model’s representational capability and generalization across diverse tasks. Experiments demonstrate that our approach reduces planning time by four orders of magnitude compared to the online planning method. Against the benchmark algorithm, it achieves 94.16% higher convergence performance, 33.54% shorter assignment path length, 51.96% lower threat value, and 40.71% faster total time. Our approach supports real-time military reconnaissance and will also facilitate rescue operations in complex cities.

Collaborative geometric reconstruction of building structures can significantly reduce communication consumption for data sharing, protect privacy, and provide support for large-scale robot application management. In recent years, geometric reconstruction of building structures has been partially studied, but there is a lack of alignment fusion studies for multi-UAV (Unmanned Aerial Vehicle)-reconstructed geometric structure models. The vertices and edges of geometric structure models are sparse, and existing methods face challenges such as low feature extraction efficiency and substantial data requirements when processing sparse graph structures after geometrization. To address these challenges, this paper proposes an efficient deep graph matching registration framework that effectively integrates interpretable feature extraction with network training. Specifically, we first extract multidimensional local properties of nodes by combining geometric features with complex network features. Next, we construct a lightweight graph neural network, named EKNet, to enhance feature representation capabilities, enabling improved performance in low-overlap registration scenarios. Finally, through feature matching and discrimination modules, we effectively eliminate incorrect pairings and enhance accuracy. Experiments demonstrate that the proposed method achieves a 27.28% improvement in registration speed compared to traditional GCN (Graph Convolutional Neural Networks) and an 80.66% increase in registration accuracy over the suboptimal method. The method exhibits strong robustness in registration for scenes with high noise and low overlap rates. Additionally, we construct a standardized geometric point cloud registration dataset.