With the wide application of unmanned aerial vehicles (UAVs), performing search and rescue missions autonomously in unknown environment has become an increasingly concerning issue. In this paper, we propose an adaptive conversion speed Q-Learning algorithm (ACSQL). Performing UAV missions autonomously is divided into two stages: rescue mission search stage and optimal path search stage. In the first stage, a UAV can find task points as soon as possible, and the efficiency of exploration is increased by adaptively adjusting the speed of the UAV. In the second stage, to get a secure and short path, we propose a subdomain search algorithm. Based on the above two stages, we improve state space and action space in reinforcement learning, and design a composite reward function, finally obtain the path of UAV to perform multiple search and rescue missions through this algorithm. In order to solve the problems of slow training convergence and high uncertainty, we initialize the Q-table by combining detection information of UAV sensors in first stage. Simulation results show that ACSQL algorithm can realize autonomous navigation and path planning of UAV in an unknown environment. Compared with traditional action space, the learning process of UAV converges faster and more stable, and it can converge in about 30 episodes. Compared with DDPG algorithm and IDWA algorithm in different scenarios, ACSQL algorithm has the shortest path length. Finally, ACSQL algorithm is verified by UAV simulator Airsim.

Rescue assignments strategy are crucial for multiple Autonomous Underwater Vehicle (multi-AUV) systems in three dimensional (3-D) complex underwater environments. Considering the requirements of rescue missions, multi-AUV systems need to be cost-effective, fast-rescuing, and less concerned about the relationship between rescue missions. The real-time rescue plays a vital role in the multi-AUV system with the characteristics mentioned above. In this paper, we propose an efficient Reward acting on Reinforcement Learning and Particle Swarm Optimization (R-RLPSO), to provide a strategy of real-time rescue assignment for the multi-AUV system in the 3-D underwater environment. This strategy consists of the following three parts. Firstly, we present a reward-based real-time rescue assignment algorithm. Secondly, we propose an Attraction Rescue Area containing a Rescue Area. For the waypoints in each Attraction Rescue Area, the reward is calculated by a linear reward function. Thirdly, to speed up the convergence of the R-RLPSO and mark the rescue states of Attraction Rescue Area and rescue area, we develop a Reward Coefficient based on the reward of all Attraction Rescue Areas and Rescue Areas. Finally, simulation results show that the system based on R-RLPSO is more cost-effective and time-saving than that of based on comparison algorithms ISOM and IACO.

Multi-UAV is a complete system represented by multiple unmanned aerial vehicles (UAV) through collaborative technology. Besides, the UAV group has the advantages in the number and functions that a single UAV does not have. In a complex environment, a single UAV cannot complete a large number of material transportations and escort functions. This paper, from the perspective of cooperative control and material escort of the UAV group, solves the escort problem in the UAV group cruise. Then, an autonomous cooperative flocking algorithm for heterogeneous UAV swarm (ACHF) is proposed. Firstly, the movement control factor of the escort UAV is proposed, and the escort UAV is deployed in the periphery of the transport UAV in the flocking process. Secondly, the system loss function including position and angle is proposed to make the escort UAV deployed as evenly as possible in the periphery of the transport UAV. Finally, in the case of different numbers of UAVs and different environments, the effectiveness of ACHF is supported by theoretical analysis and demonstrated by experiments. The results show that the escort UAVs can be evenly distributed in the periphery of the transport UAVs, forming a group of UAVs with autonomous cooperative capability.

The problem of task allocation of UAV systems is a hot issue in scientific research. Aiming at overcome the shortcomings of existing algorithms in terms of load balancing and execution efficiency, an algorithm Improved Self-Organizing Mapping (ISOM) is proposed in this paper. Firstly, considering the two factors of the UAV flight distance and the mission execution time, the load balance of the UAV is designed. The sum of the track lengths of multiple UAVs and the total time required by multiple UAVs to complete all the tasks in the task area are used to evaluate the quality of cooperative task assignment of multiple UAVs. Secondly, the learning rate and neighborhood function of nonlinear change are designed to ensure the stability and accelerate its convergence. With the increase of the number of iterations, the radius of the neighborhood gradually decreases, and the renewal range of the superior neighborhood is determined by the nonlinear function. Finally, in order to solve the problem of new tasks of UAV and UAV failure, the attention mechanism based on the ISOM algorithm is introduced. Different attention is allocated by the distance between the UAV and the mission points to ensure the missions can be fully executed. Compared with algorithms PSO-GA, Gurobi and ORTools, the time to complete the task is effectively reduced, respectively. At the same time, the algorithm is verified in a large task environment. When the number of tasks is 200 and the number of UAVs is 8, in the TSPLIB task set, the simulation results demonstrate the high efficiency and flexibility of the proposed algorithm.

With the further development of Unmanned Aerial Vehicle (UAV) technologies, research on multi-UAV formations have also received more attention. Unmanned Aerial Vehicles (UAVs) cooperate with each other to form a formation group, which can give full play to the advantages that a single UAV does not have, and more capable of working in multi-task scenarios. Based on the Sierpinski fractal structure, this paper proposes a hybrid formation control architecture. To address the aggregation problem of UAV swarms, a multi-agent deep reinforcement learning (MADRL) method is used for aggregation control. To improve the efficiency, accuracy, effectiveness and scalability of MADRL in environments with a large number of UAVs, a multi-intelligent deep deterministic policy gradient method based on attention mechanism and adaptive accuracy (3A-MADDPG) is proposed. The method enables each agent to selectively focus on the information of other agents, with adaptive learning rate to dynamically learn its own critic network. The algorithm has a large learning rate in the early stage and converges quickly. The later stage of the algorithm has small learning rate and high accuracy, and combined with the fixed variable hybrid reward function designed in the paper, so the UAV can fly to the center of the expected region more accurately, making the whole cluster more accurate. Experiments show that the 3A algorithm proposed in the paper converges faster and achieves higher accuracy than the benchmark algorithm, whether it is the cluster aggregation formation or the separation and aggregation of sub-formations.

Energy optimization is a critical issue in three-dimensional (3D) underwater acoustic sensor networks (UASNs). Intelligent path planning can be applied to extend the lifetime of autonomous underwater vehicles (AUVs) which has attracted many researchers' attention as a key component of UASNs in recent years. In this paper, we put forward an algorithm of distance evolution nonlinear particle swarm optimization (DENPSO), aiming at finding an energy-efficient stable path for AUVs in 3D UASNs. First, in order to ensure that the particles fully explore the 3D underwater environment during the evolution process, we convert the inertia weighting factor and learning factor from linearity to nonlinearity. Second, to avoid particles falling into local optimum regions, the particles of the poor search regions are randomly perturbed by the distance evolution factor. Third, we apply the penalty function to describe the energy optimization goal under the obstacles and ocean currents. To quantify the role of obstacle avoidance in the penalty function, each path is divided into several micro-element points based on the cubic spline interpolation method. Then, we propose a degree value factor to measure the micro-element points falling within the obstacle coverage regions. Finally, simulations are finished in 3D underwater environment and the real environment based on regional ocean model system (ROMS). The results show that DENPSO can avoid the obstacles along the eddy current direction, where the energy consumption of algorithm DENPSO is, respectively, reduced by 2.1514e+03 J and 1.049e+07 J compared with the algorithm LPSO in the above-mentioned environment.