This paper presents a reinforcement learning (RL)-based methodology for the online fine-tuning of PD controller gains, with the goal of bridging the gap between simulation-trained controllers and real-world quadrotor applications. As a first step toward real-world implementation, the proposed approach applies a Deep Deterministic Policy Gradient (DDPG) algorithm—an off-policy actor–critic method—to adjust the gains of a quadrotor attitude PD controller during flight. The RL agent was initially trained offline in a simulated environment, using MATLAB/Simulink 2024a and the UAV Toolbox Support Package for PX4 Autopilots v1.14.0. The trained controller was then validated through both simulation and experimental flight tests. Comparative performance analyses were conducted between the hand-tuned and RL-tuned controllers. Our results demonstrate that the RL-based tuning method successfully adapts the controller gains in real time, leading to improved attitude tracking and reduced steady-state error. This study constitutes the first stage of a broader research effort investigating RL-based PID, LQR, MRAC, and Koopman-integrated RL-based PID controllers for real-time quadrotor control.
13 February 2024 in IEEE Transactions on Aerospace and Electronic Systems
Mars exploration has recently witnessed major interest within the scientific community, particularly because unmanned aerial robotic platforms offer reliable alternatives for acquiring and collecting data and information from the Red Planet. However, the specific conditions of the Martian environment result in a restricted flight envelope when flying close to Mars and then landing on the surface of Mars. Therefore, in addition to the requirement to develop an aerial platform suitable for operations on Mars, autonomous navigation strategies and robust controllers are also needed for exploration tasks. It is argued that hexacopters with their relatively compact design represent a promising solution for autonomous exploration tasks on Mars, overcoming at the same time the limitations of wheel-based rovers. This research focuses on the design of a Mars hexacopter for a scouting mission in the Jezero region of Mars. The configuration and architecture of the hexacopter follow NASA conceptual study of the Mars science helicopter. Then, the mission profile for mapping the Belva crater is examined, followed by a detailed approach to implement and test observer-based navigation and control strategies. A comprehensive simulated experiments environment based on the integration of robot operating system and Ardupilot is also presented, used to validate the overall system architecture and mission parameters considering both the morphological shape of the explored crater and the atmospheric conditions of Mars.
24 January 2024 in Journal of Intelligent & Robotic Systems
This technical note proves analytically how the exact equivalence of the Newton-Euler and Euler-Lagrange modeling formulations as applied to multirotor UAVs is achieved. This is done by deriving a correct Euler-Lagrange multirotor attitude dynamics model. A review of the published literature reveals that the commonly adopted Euler-Lagrange multirotor dynamics model is equivalent to the Newton-Euler model only when it comes to the position dynamics, but not in the attitude dynamics. Step-by-step derivations and calculations are provided to show how modeling equivalence to the Newton-Euler formulation is proven. The modeling equivalence is then verified by obtaining identical results in numerical simulation studies. Simulation results also illustrate that when using the correct model for feedback linearization, controller stability at high gains is improved.
A reinforcement learning (RL) based methodology is proposed and implemented for online fine-tuning of PID controller gains, thus, improving quadrotor effective and accurate trajectory tracking. The RL agent is first trained offline on a quadrotor PID attitude controller and then validated through simulations and experimental flights. RL exploits a Deep Deterministic Policy Gradient (DDPG) algorithm, which is an off-policy actor-critic method. Training and simulation studies are performed using Matlab/Simulink and the UAV Toolbox Support Package for PX4 Autopilots. Performance evaluation and comparison studies are performed between the hand-tuned and RL-based tuned approaches. The results show that the controller parameters based on RL are adjusted during flights, achieving the smallest attitude errors, thus significantly improving attitude tracking performance compared to the hand-tuned approach.
11 November 2026 in Journal of Intelligent & Robotic Systems
Unmanned Aerial Vehicles (UAVs) are increasingly deployed across civilian and military domains due to their mobility, flexibility, and low energy consumption. Traditional UAV path-planning and navigation techniques often suffer from high computational cost, limited adaptability, and suboptimal path generation, motivating the use of learning-based solutions. Deep Reinforcement Learning (DRL) enables UAVs to learn optimal navigation strategies directly from interactions with the environment, handling continuous and high-dimensional action spaces typical of UAV dynamics. However, conventional DRL methods struggle to capture long-term temporal dependencies. This work presents a goal-conditioned DRL framework for multirotor navigation toward a target destination, enhanced with a Transformer-encoder and a Long Short-Term Memory (LSTM) layer. Particularly, both architectures address the DRL limitations by modeling long-range dependencies through a self-attention mechanism and preserving temporal dependencies. The framework integrates two stable, probabilistic DRL algorithms, PPO and A2C. Compared with their standalone counterparts, the Transformer-enhanced architectures improve validation stability and trajectory consistency, with the most pronounced improvement observed for PPO. Furthermore, a novel reward function for continuous action spaces is proposed, combining multiple contributions with a physics-inspired term based on the Least Action Principle (LAP), which encourages efficient and dynamically consistent flight. A simulation pipeline is developed to evaluate the approach. Training is first performed on a quadrotor dynamic model, followed by Software-in-the-Loop (SIL) on PX4 and Gazebo validation and Hardware-in-the-Loop (HIL) deployment on a Raspberry-Pi (RPI). The proposed pipeline is designed to facilitate model prototyping while supporting a structured progression toward real-world flight testing. The results obtained from the simulated experiments show that the Transformer-enhanced architecture improves trajectory robustness compared to baseline DRL policies, while the proposed reward formulation promotes smoother and more action-consistent flight behavior. These findings demonstrate the effectiveness of combining transformer-based temporal modeling with physics-informed reward design for autonomous UAV navigation.