A safe and reliable task planning method is a prerequisite for the collaborative execution of ocean observation data collection tasks by multiple unmanned surface vessels (multi-USVs). Deep Reinforcement Learning (DRL) combines the powerful nonlinear function-fitting capabilities of deep neural networks with the decision-making and control abilities of reinforcement learning, providing a novel approach to solving the multi-USV task planning problem. However, when applied to the field of multi-USV task planning, it faces challenges such as a vast exploration space, extended training times, and unstable training process. To this end, this paper proposes a multi-USV task planning method based on improved deep reinforcement learning. The proposed method draws on the idea of a value decomposition network, breaking down the multi-USV task planning problem into two subproblems: task allocation and autonomous collision avoidance. Different state spaces, action spaces, and reward functions are designed for the various subproblems. Based on this, a deep neural network is used to map the state space of each subproblem to the action space of each USV, and the generated strategy of the deep neural network is assessed based on the corresponding reward function. This successfully integrates task allocation and path planning into a comprehensive task planning framework. Deep neural networks consist of the Actor networks and the Critic networks. During the training phase of the Critic network, different methods are used to train different Critic networks to improve the convergence speed of the algorithm. An improved temporal difference error method is specifically applied to train the Critic network for evaluating autonomous collision avoidance strategies, resulting in improving the autonomous collision avoidance ability of USVs. At the same time, to improve the efficiency of task allocation, hierarchical mechanisms, and regional division mechanisms are introduced to construct sub-system task planning models, which further decompose the task planning problem. A combination of successor features and an improved temporal difference error method is specifically applied to train another Critic network for evaluating the sub-systems task allocation schemes and collaborative motion trajectories, aiming to enhance the allocation efficiency of the sub-systems. Furthermore, transfer learning is employed to merge the sub-system task planning, using it as a constraint to direct the exploration and assessment of both the cluster task allocation schemes and the cluster collaborative motion trajectories. This enables rapid and accurate learning for task allocation within the multi-USV cluster. During the training phase of the Actor network, the introduction of the experience replay method and target network technique is employed to enhance the proximal policy optimization algorithm. This facilitates distributed joint training of the Actor network, thereby improving the accuracy of the algorithm. Simulation results validate the effectiveness and superiority of this method.

Focusing on the collision avoidance problem for Unmanned Surface Vehicles (USVs) in the scenario of multi-vessel encounters, a USV autonomous obstacle avoidance algorithm based on the improved velocity obstacle method is proposed. The algorithm is composed of two parts: a multi-vessel encounter collision detection model and a path re-planning algorithm. The multi-vessel encounter collision detection model draws on the idea of the velocity obstacle method through the integration of characteristics such as the USV dynamic model in the marine environment, the encountering vessel motion model, and the International Regulations for Preventing Collisions at Sea (COLREGS) to obtain the velocity obstacle region in the scenario of USV and multi-vessel encounters. On this basis, two constraint conditions for the motion state space of USV obstacle avoidance behavior and the velocity obstacle region are added to the dynamic window algorithm to complete a USV collision risk assessment and generate a collision avoidance strategy set. The path re-planning algorithm is based on the premise of the minimum resource cost and uses an improved particle swarm algorithm to obtain the optimal USV control strategy in the collision avoidance strategy set and complete USV path re-planning. Simulation results show that the algorithm can enable USVs to safely evade multiple short-range dynamic targets under COLREGS.

Unmanned surface vehicles (USVs) serve as crucial data acquisition nodes within the ocean observation network, enabling the collection of various types of data from the ocean-atmosphere interface and below the sea surface. However, USVs face environmental challenges such as tides, ocean currents, and winds in complex marine environments. These factors introduce uncertainty into task decision-making and motion control, significantly affecting the real-time operation, stability, and effectiveness of the USV system. This paper proposes an asynchronous multithreading reinforcement control decision method to address these issues for USVs. By establishing a task environment model, this method captures and analyzes the dynamic characteristics of the marine environment while decomposing the tasks accordingly. Based on this, the method enables collaborative control for path planning, emergency collision avoidance, and path tracking by mapping the state space of each task to the action space of the USV using a deep neural network. To improve the real-time performance of this method, experience replay is integrated with target network technology, which strengthens the proximal policy optimization algorithm under the asynchronous multithreading framework. To enhance the method’s accuracy, effective reward functions are designed based on task objectives and constraints. These functions evaluate the strategies generated by the deep neural network, ensuring that the USV always adopts optimal strategies. The simulation results validate the effectiveness and superiority of the proposed method, highlighting its substantial potential for engineering applications.

To enable collaborative reconnaissance/strike/assessment of underwater time-sensitive targets by heterogeneous multi-agent systems, this study proposes a heterogeneous multi-agent collaborative decision-making method based on deep reinforcement learning (DRL). The method integrates two core frameworks: a heterogeneous multi-agent task allocation learning framework and a single-agent multi-task learning framework. The task allocation framework combines the Proximal Policy Optimization (PPO) algorithm with experience replay to train the Actor network, ensuring stable iterative updates of task allocation policies toward high-reward directions. Concurrently, the Critic network is trained using successor features and Temporal Difference (TD) methods, establishing a foundation for transfer learning. A region partitioning mechanism is introduced to construct a foundational knowledge base, enabling the transfer of sub-region knowledge acquired by multi-agents to target regions, thereby addressing complex underwater task allocation scenarios. The single-agent multi-task learning framework employs an experience replay pool with knowledge transfer attributes and policy distillation technology, allowing each agent to assimilate task-specific expertise from heterogeneous peers across diverse mission scenarios. This capability supports multi-task execution, including path planning, emergency obstacle avoidance, and trajectory tracking.

To address the challenge of collision avoidance for Unmanned Surface Vehicles (USVs) under highly dynamic spatiotemporal marine environments, this paper proposes a USV path planning method that integrates the Velocity Obstacle (VO) approach with an improved Multi-Objective Particle Swarm Optimization (MOPSO) algorithm. The method constructs a velocity obstacle area by incorporating environmental disturbances, USV dynamics, encounter scenarios, and the International Regulations for Preventing Collisions at Sea (COLREGS), thereby generating a set of feasible avoidance strategies. Based on this, a multi-objective path planning model is established with the dual optimization goals of minimizing energy consumption and collision risk. To overcome the limitations of traditional MOPSO, such as poor adaptability and susceptibility to local optima, this paper introduces improvements in particle encoding, update strategy, and external archive mechanism to enhance convergence and solution diversity. Simulation results demonstrate that the proposed method can provide safe and energy-efficient autonomous collision avoidance paths for USVs in complex dynamic marine environments, highlighting its practicality and intelligence.