An autonomous vehicle (AV) requires an accurate perception of its surrounding environment to operate reliably. The perception system of an AV, which normally employs machine learning (e.g., deep learning), transforms sensory data into semantic information that enables autonomous driving. Object detection is a fundamental function of this perception system, which has been tackled by several works, most of them using 2D detection methods. However, the 2D methods do not provide depth information, which is required for driving tasks, such as path planning, collision avoidance, and so on. Alternatively, the 3D object detection methods introduce a third dimension that reveals more detailed object's size and location information. Nonetheless, the detection accuracy of such methods needs to be improved. To the best of our knowledge, this is the first survey on 3D object detection methods used for autonomous driving applications. This paper presents an overview of 3D object detection methods and prevalently used sensors and datasets in AVs. It then discusses and categorizes the recent works based on sensors modalities into monocular, point cloud-based, and fusion methods. We then summarize the results of the surveyed works and identify the research gaps and future research directions.

For an autonomous vehicle to operate safely and effectively, an accurate and robust localization system is essential. While there are a variety of vehicle localization techniques in literature, there is a lack of effort in comparing these techniques and identifying their potentials and limitations for autonomous vehicle applications. Hence, this paper evaluates the state-of-the-art vehicle localization techniques and investigates their applicability on autonomous vehicles. The analysis starts with discussing the techniques which merely use the information obtained from on-board vehicle sensors. It is shown that although some techniques can achieve the accuracy required for autonomous driving but suffer from the high cost of the sensors and also sensor performance limitations in different driving scenarios (e.g., cornering and intersections) and different environmental conditions (e.g., darkness and snow). This paper continues the analysis with considering the techniques which benefit from off-board information obtained from V2X communication channels, in addition to vehicle sensory information. The analysis shows that augmenting off-board information to sensory information has potential to design low-cost localization systems with high accuracy and robustness, however, their performance depends on penetration rate of nearby connected vehicles or infrastructure and the quality of network service.

Behaviour prediction function of an autonomous vehicle predicts the future states of the nearby vehicles based on the current and past observations of the surrounding environment. This helps enhance their awareness of the imminent hazards. However, conventional behavior prediction solutions are applicable in simple driving scenarios that require short prediction horizons. Most recently, deep learning-based approaches have become popular due to their promising performance in more complex environments compared to the conventional approaches. Motivated by this increased popularity, we provide a comprehensive review of the state-of-the-art of deep learning-based approaches for vehicle behavior prediction in this article. We firstly give an overview of the generic problem of vehicle behavior prediction and discuss its challenges, followed by classification and review of the most recent deep learning-based solutions based on three criteria: input representation, output type, and prediction method. The article also discusses the performance of several well-known solutions, identifies the research gaps in the literature and outlines potential new research directions.

3D object detection is a common function within the perception system of an autonomous vehicle and outputs a list of 3D bounding boxes around objects of interest. Various 3D object detection methods have relied on fusion of different sensor modalities to overcome limitations of individual sensors. However, occlusion, limited field-of-view and low-point density of the sensor data cannot be reliably and cost-effectively addressed by multi-modal sensing from a single point of view. Alternatively, cooperative perception incorporates information from spatially diverse sensors distributed around the environment as a way to mitigate these limitations. This article proposes two schemes for cooperative 3D object detection using single modality sensors. The early fusion scheme combines point clouds from multiple spatially diverse sensing points of view before detection. In contrast, the late fusion scheme fuses the independently detected bounding boxes from multiple spatially diverse sensors. We evaluate the performance of both schemes, and their hybrid combination, using a synthetic cooperative dataset created in two complex driving scenarios, a T-junction and a roundabout. The evaluation shows that the early fusion approach outperforms late fusion by a significant margin at the cost of higher communication bandwidth. The results demonstrate that cooperative perception can recall more than 95% of the objects as opposed to 30% for single-point sensing in the most challenging scenario. To provide practical insights into the deployment of such system, we report how the number of sensors and their configuration impact the detection performance of the system.

The state-of-the-art decision and planning approaches for autonomous vehicles have moved away from manually designed systems, instead focusing on the utilisation of large-scale datasets of expert demonstration via Imitation Learning (IL). In this paper, we present a comprehensive review of IL approaches, primarily for the paradigm of end-to-end based systems in autonomous vehicles. We classify the literature into three distinct categories: 1) Behavioural Cloning (BC), 2) Direct Policy Learning (DPL) and 3) Inverse Reinforcement Learning (IRL). For each of these categories, the current state-of-the-art literature is comprehensively reviewed and summarised, with future directions of research identified to facilitate the development of imitation learning based systems for end-to-end autonomous vehicles. Due to the data-intensive nature of deep learning techniques, currently available datasets and simulators for end-to-end autonomous driving are also reviewed.

Automated vehicles are increasingly getting main-streamed and this has pushed development of systems for autonomous manoeuvring (e.g., lane-change, merge, and overtake) to the forefront. A novel framework for situational awareness and trajectory planning to perform autonomous overtaking in high-speed structured environments (e.g., highway and motorway) is presented in this paper. A combination of a potential field like function and reachability sets of a vehicle are used to identify safe zones on a road that the vehicle can navigate towards. These safe zones are provided to a tube-based robust model predictive controller as reference to generate feasible trajectories for combined lateral and longitudinal motion of a vehicle. The strengths of the proposed framework are: 1) it is free from non-convex collision avoidance constraints; 2) it ensures feasibility of trajectory even if decelerating or accelerating while performing lateral motion; and 3) it is real-time implementable. The ability of the proposed framework to plan feasible trajectories for high-speed overtaking is validated in a high-fidelity IPG CarMaker and Simulink co-simulation environment.