The marine environment is a hostile setting for robotics. It is strongly unstructured, uncertain, and includes many external disturbances that cannot be easily predicted or modeled. In this work, we attempt to control an autonomous underwater vehicle (AUV) to perform a waypoint tracking task, using a machine learning-based controller. There has been great progress in machine learning (in many different domains) in recent years; in the subfield of deep reinforcement learning, several algorithms suitable for the continuous control of dynamical systems have been designed. We implemented the soft actor–critic (SAC) algorithm, an entropy-regularized deep reinforcement learning algorithm that allows fulfilling a learning task and encourages the exploration of the environment simultaneously. We compared a SAC-based controller with a proportional integral derivative (PID) controller on a waypoint tracking task using specific performance metrics. All tests were simulated via the UUV simulator. We applied these two controllers to the RexROV 2, a six degrees of freedom cube-shaped remotely operated underwater Vehicle (ROV) converted in an AUV. We propose several interesting contributions as a result of these tests, such as making the SAC control and guiding the AUV simultaneously, outperforming the PID controller in terms of energy saving, and reducing the amount of information needed by the SAC algorithm inputs. Moreover, our implementation of this controller allows facilitating the transfer towards real-world robots. The code corresponding to this work is available on GitHub.

Learning-based adaptive control methods hold the potential to empower autonomous agents in mitigating the impact of process variations with minimal human intervention. However, their application to autonomous underwater vehicles (AUVs) has been constrained by two main challenges: (1) the presence of unknown dynamics in the form of sea current disturbances, which cannot be modelled or measured due to limited sensor capability, particularly on smaller low-cost AUVs, and (2) the nonlinearity of AUV tasks, where the controller response at certain operating points must be excessively conservative to meet specifications at other points. Deep Reinforcement Learning (DRL) offers a solution to these challenges by training versatile neural network policies. Nevertheless, the application of DRL algorithms to AUVs has been predominantly limited to simulated environments due to their inherent high sample complexity and the distribution shift problem. This paper introduces a novel approach by combining the Maximum Entropy Deep Reinforcement Learning framework with a classic model-based control architecture to formulate an adaptive controller. In this framework, we propose a Sim-to-Real transfer strategy, incorporating a bio-inspired experience replay mechanism, an enhanced domain randomisation technique, and an evaluation protocol executed on a physical platform. Our experimental assessments demonstrate the effectiveness of this method in learning proficient policies from suboptimal simulated models of the AUV. When transferred to a real-world vehicle, the approach exhibits a control performance three times higher compared to its model-based nonadaptive but optimal counterpart.

Despite technological growth and worldwide advancements in various fields, the agriculture sector continues to face numerous challenges such as desertification, environmental pollution, resource scarcity, and the excessive use of pesticides and inorganic fertilizers. These unsustainable problems in agricultural field can lead to land degradation, threaten food security, affect the economy, and put human health at risk. To mitigate these global issues, it is essential for researchers and agricultural professionals to promote advancements in smart agriculture by integrating modern technologies such as Internet of Things (IoT), Unmanned Aerial Vehicles (UAVs), Wireless Sensor Networks (WSNs), and more. Among these technologies, this paper focuses on UAVs, particularly quadcopters, which can assist in each phase of the agricultural cycle and improve productivity, quality, and sustainability. With their diverse capabilities, quadcopters have become the most widely used UAVs in smart agriculture and are frequently utilized by researchers in various projects. To explore the different aspects of quadcopters’ use in smart agriculture, this paper focuses on the following: (a) the unique advantages of quadcopters over other UAVs, including an examination of the quadcopter types particularly used in smart agriculture; (b) various agricultural missions where quadcopters are deployed, with examples highlighting their indispensable role; (c) the modelling of quadcopters, from configurations to the derivation of mathematical equations, to create a well-modelled system that closely represents real-world conditions; and (d) the challenges that must be addressed, along with suggestions for future research to ensure sustainable development. Although the use of UAVs in smart agriculture has been discussed in other papers, to the best of our knowledge, none have specifically examined the most popular among them, “quadcopters”, and their particular use in smart agriculture in terms of types, applications, and modelling techniques. Therefore, this paper provides a comprehensive survey of quadcopters’ use in smart agriculture and offers researchers and engineers valuable insights into this evolving field, presenting a roadmap for future enhancements and developments.

Deep Reinforcement Learning (DRL) methods are increasingly being applied in Unmanned Underwater Vehicles (UUV) providing adaptive control responses to environmental disturbances. However, in physical platforms, these methods are hindered by their inherent data inefficiency and performance degradation when subjected to unforeseen process variations. This is particularly notorious in UUV manoeuvring tasks, where process observability is limited due to the complex dynamics of the environment in which these vehicles operate. To overcome these limitations, this paper proposes a novel Biologically-Inspired Experience Replay method (BIER), which considers two types of memory buffers: one that uses incomplete (but recent) trajectories of state-action pairs, and another that emphasises positive rewards. The BIER method’s ability to generalise was assessed by training neural network controllers for tasks such as inverted pendulum stabilisation, hopping, walking, and simulating halfcheetah running from the Gym-based Mujoco continuous control benchmark. BIER was then used with the Soft Actor-Critic (SAC) method on UUV manoeuvring tasks to stabilise the vehicle at a given velocity and pose under unknown environment dynamics. The proposed method was evaluated through simulated scenarios in a ROS-based UUV Simulator, progressively increasing in complexity. These scenarios varied in terms of target velocity values and the intensity of current disturbances. The results showed that BIER outperformed standard Experience Replay (ER) methods, achieving optimal performance twice as fast as the latter in the assumed UUV domain.