In this article, we aim to design a task scheduling scheme for the underwater multi-objective task of target hunting and environmental search. A distributed autonomous underwater vehicle (AUV) network is deployed to perform the task, where AUVs equipped with sensors can cooperatively search the environment and hunt the target by sharing local information. To achieve efficient exploration of the overall environment by the AUV network, we design an intra-network cooperative searching approach based on the age of information (AoI). Besides, it is critical to conceive an energy-efficient mechanism due to the energy constraints of AUVs and the difficulty of sustainable energy supply. To address the aforementioned issues, we propose an energy-efficient distributed multi-agent proximal policy optimization (DMAPPO) scheme to perform real-time AUV target hunting and environment searching in underwater turbulent fields. The proposed scheme can adjust the number of AUVs assigned to each objective according to practical requirement and residual energy. Distributed AUVs can make decisions autonomously and cooperatively complete the task efficiently through limited information interaction. In addition, we derive a lower bound on the policy improvement of MAPPO. Moreover, our simulation results demonstrate that the proposed scheme outperforms the standard algorithms in terms of hunting efficiency, degree of searching, and network energy efficiency.

As an essential part of the 6G sea-land-air integrated network, underwater networking has attracted increasing attention and has been widely studied. The key for improving its performance is the communication optimization based on data rate, throughput, latency, reliability, spectrum utilization, and other factors impacting on the quality of service (QoS). However, the poor underwater communication environment makes it difficult to improve the communication quality of underwater networking and brings many challenges to the design of optimization schemes. In the face of complex and unknown dynamic underwater environment, the optimization schemes need to have a higher level of adaptability and intelligence, so as to carry out autonomous decision-making and multi-objective optimization under different conditions. To meet the above challenges and needs, reinforcement learning (RL) is widely used to obtain the optimal strategy for underwater communication. Nevertheless, there is still a lack of comprehensive reviews on using RL to optimize underwater communication networking. Therefore, this survey comprehensively investigates the application of RL in underwater networking to guide the optimization of underwater communication in the future and bridge this gap. Specifically, we provide an overview of RL usage processes and tools and detail its various applications in underwater communication networking, including spectrum resource allocation and development, throughput improvement and delay reduction, reliability improvement, energy saving, and energy efficiency optimization, data sensing and processing, and intelligent cluster networking. Based on the review, we further analyze the open challenges and research directions of RL-enabled underwater communication networking in the future.

In recent years, unmanned aerial vehicles (UAVs) have been widely used in ocean target tracking and image acquisition for processing. Due to the limited energy of the UAV and the high computational complexity associated with image processing tasks, a lightweight energy-saving target tracking scheme is designed for the UAV, and the unmanned surface vehicle (USV) based mobile edge computing (MEC) networks are adopted to share the computing load of the UAV. Due to the randomness of the environment, we formulate data processing, computation offloading, resource allocation, and target-tracking as a joint stochastic optimization problem. This paper investigates a two-stage optimization scheme to address the problem. First, we employ a Lyapunov-based approach to convert the stochastic optimization problem into a deterministic per-time slot problem under communication and computing resources constraints. Then, we develop a real-time target tracking scheme for the UAV based on the Elman neural network. Numerical results validate that the designed tracking scheme can effectively minimize propulsion energy consumption while maintaining a high success rate in tracking. Furthermore, the proposed method balances data-related energy consumption, image detection accuracy, and stability of the data storage queue.

In recent years, the Internet of Underwater Things (IoUT) has garnered significant attention owing to its potential in ocean exploration and monitoring. However, environmental erosion and limited energy can cause node failures, leading to routing voids, communication congestion, and even IoUT breakdowns. Addressing these challenges, this work considers a node repair scheme for multiple autonomous underwater vehicles (AUVs) to search and repair faulty nodes to ensure the stable operation of the IoUT networks. Moreover, AUVs should adapt automatically to the unknown environment, working in cooperative or separative modes to balance repair efficiency and coverage. We propose a multiagent reinforcement learning-based AUV-assisted node repair (RANR) scheme, which considers limited underwater communication and scheduling between AUVs. To further enhance work efficiency, we introduce area information entropy to reduce redundant coverage among AUVs. Simulation results demonstrate that the RANR scheme is highly applicable to different working conditions.

Sensing and communication sharing the same hardware and frequency band has become a key technology for future 5G/6G systems to achieve high-performance integrated sensing and communication (ISAC), which has attracted a great deal of attention in both academia and industry. In this letter, a device-free ISAC system based on distributed unmanned aerial vehicles (UAVs) is designed to realize the sensing of vehicles on urban roads and transmit information to base stations through wireless communication. In order to realize autonomous decision-making of UAVs and solve the problem of stochastic optimization and multi-index tradeoffs caused by a time-varying environment, we propose an algorithm based on multi-agent reinforcement learning (MARL). The simulation results verify that the proposed scheme can significantly improve the sensing performance of the UAVs and ensure communication quality with base stations. Besides, we realize the sensing fairness for users, the flight safety of UAVs, and can save energy through hyperparameter tuning.

Autonomous underwater vehicles (AUVs) can function as sensor nodes in Internet of Underwater Things (IoUT), contributing to ocean exploration and monitoring by collecting and transmitting data to the base station. Most of the routing algorithms applied to IoUT require the participation of stationary nodes and seldom consider the fluctuations of network topology, which cannot be directly applied to the IoUT composed of AUVs. The focus of this research is to examine the problem of packet routing in a dynamic AUV-assisted IoUT, with the ultimate goal of ensuring the effective transmission of underwater information. We analyze the transmission pattern of underwater acoustic signals and the consequent communication disruption between AUVs, which helps establish the Age of Information (AoI) and bit error rate (BER) of data through modeling. A routing algorithm that utilizes the branch-and-bound (BB) technique has been suggested, alongside the introduction of the Value of information (VoI) to enable the joint optimization of the AoI and BER. We describe two nearly optimal heuristic algorithms for networks with a high number of AUVs. The AFA-ACO-BB strategy is designed based on the above algorithms and the influence of AUV motion on link reliability is considered. Moreover, we have developed a power regulation mechanism that can effectively minimize the occurrence of network packet loss and energy waste. The simulation results demonstrate that the proposed scheme outperforms certain classically related schemes in terms of AoI and BER, while simultaneously maintaining superior packet loss rate (PLR) and energy consumption.