In this paper, we propose an autonomous UAV path planning framework using deep reinforcement learning approach. The objective is to employ a self-trained UAV as a flying mobile unit to reach spatially distributed moving or static targets in a given three dimensional urban area. In this approach, a Deep Deterministic Policy Gradient (DDPG) with continuous action space is designed to train the UAV to navigate through or over the obstacles to reach its assigned target. A customized reward function is developed to minimize the distance separating the UAV and its destination while penalizing collisions. Numerical simulations investigate the behavior of the UAV in learning the environment and autonomously determining trajectories for different selected scenarios.
Nowadays, Wireless Sensor Networks (WSNs) are playing a vital and sustainable role in many verticals touching different aspects of our lives including civil, public, and military applications. WSNs majorly consist of a few to several sensor nodes, that are connected to each other via wireless communication links and require real-time or delayed data transfer. In this paper, we propose an autonomous Unmanned Aerial Vehicle (UAV)-enabled data gathering mechanism for delay-tolerant WSN applications. The objective is to employ a self-trained UAV as a flying mobile unit collecting data from ground sensor nodes spatially distributed in a given geographical area during a predefined period of time. In this approach, two Reinforcement Learning (RL) approaches, specifically Deep Deterministic Gradient Decent (DDPG) and Q-learning (QL) algorithms, are jointly employed to train the UAV to understand the environment and provide effective scheduling to accomplish its data collection mission. The DDPG is used to autonomously decide the best trajectory to adopt in an obstacle-constrained environment, while the QL is developed to determine the order of nodes to visit such that the data collection time is minimized. The schedule is obtained while considering the limited battery capacity of the flying unit, its need to return the charging station, the time windows of data acquisition, and the priority of certain sensor nodes. Customized reward functions are designed for each RL model and, through numerical simulations, we investigate their training performances. We also analyze the behavior of the autonomous UAV for different selected scenarios and corroborate the ability of the proposed approach in performing effective data collection. A comparison with the deterministic optimal solution is provided to validate the performance of the learning-based approach.
Micro-unmanned aerial vehicles (UAVs), also known as drones, have been recognized as an emerging technology offering a plethora of applications touching various aspects of our lives, such as surveillance, agriculture, entertainment, and intelligent transportation systems (ITS). Furthermore, due to their low cost and ability to be fitted with transmitters, cameras, and other on-board sensors, UAVs can be seen as potential flying Internet-of-things (IoT) devices interconnecting with their environment and allowing for more mobile flexibility in the network. This paper overviews the beneficial applications that UAVs can offer to smart cities, and particularly to ITS, while highlighting the main challenges that can be encountered. Afterward, it proposes several potential solutions to organize the operation of UAV swarms, while addressing one of their main issues: their battery-limited capacity. Finally, open research areas that should be undertaken to strengthen the case for UAVs to become part of the smart infrastructure for futuristic cities are discussed.
9 March 2020 in IEEE Open Journal of Vehicular Technology
Considerable attention has been given to leverage a variety of smart city applications using unmanned aerial vehicles (UAVs). The rapid advances in artificial intelligence can empower UAVs with autonomous capabilities allowing them to learn from their surrounding environment and act accordingly without human intervention. In this paper, we propose a spatiotemporal scheduling framework for autonomous UAVs using reinforcement learning. The framework enables UAVs to autonomously determine their schedules to cover the maximum of pre-scheduled events spatially and temporally distributed in a given geographical area and over a pre-determined time horizon. The designed framework has the ability to update the planned schedules in case of unexpected emergency events. The UAVs are trained using the Q-learning (QL) algorithm to find effective scheduling plan. A customized reward function is developed to consider several constraints especially the limited battery capacity of the flying units, the time windows of events, and the delays caused by the UAV navigation between events. Numerical simulations show the behavior of the autonomous UAVs for various scenarios and corroborate the ability of QL to handle complex vehicle routing problems with several constraints. A comparison with an optimal deterministic solution is also provided to validate the performance of the learning-based solution.
In this paper, we design a routing scheduling framework for multi-task agent using reinforcement learning. The objective is to employ an autonomous agent to cover the maximum of pre-scheduled tasks spatially and temporally distributed in a given geographical area over a pre-determined period of time. In this approach, we train the agent using Q-learning (QL), an off-policy temporal difference learning algorithm, that finds effective near-optimal solutions. The agent uses the feedback received from previously taken decisions to learn and adapt its next actions accordingly. A customized reward function was developed to consider the time windows of task and the delays caused by agent navigation between tasks. Numerical simulations show the behavior of the autonomous agent for different selected scenarios and corroborate the ability of QL to handle complex vehicle routing problems with several constraints.
Accurate and timely occupancy prediction has the potential to improve the efficiency of energy management systems in smart buildings. Occupancy prediction heavily depends on historical occupancy-related data collected from various sensor sources. Unfortunately, a major problem in that context is the difficulty to collect training data. This situation inspired us to rethink the occupancy prediction problem, proposing the use of an original principled approach based on occupancy estimation via interactive learning to collect the needed training data. Following that, the collected data, along with various features, were fed into several algorithms to predict future occupancy. This paper mainly proposes a weakly supervised occupancy prediction framework based on office sensor readings and occupancy estimations derived from an interactive learning approach. Two studies are the main emphasis of this paper. The first is the prediction of three occupancy states, referred to as discrete states: absence, presence of one occupant, and presence of more than one occupant. The purpose of the second study is to anticipate the future number of occupants, i.e., continuous states. Extensive simulations were run to demonstrate the merits of the proposed prediction framework’s performance and to validate the interactive learning-based approach’s ability to contribute to the achievement of effective occupancy prediction. The results reveal that LightGBM, a machine learning model, is a better fit for short-term predictions than known recursive neural networks when dealing with a limited dataset. For a 24 h window forecast, LightGBM improved accuracy from 38% to 50%, which is an excellent result for non-aggregated data (single office).