Developing autonomous driving technologies necessitates addressing safety and cost concerns. Both academic research and commercial applications of autonomous driving vehicles require extensive simulation and real-world testing. The challenge lies in effectively transferring driving knowledge from the virtual simulation world to the reality world, known as the reality gap (RG). This gap arises due to differences in lighting, textures, vehicle dynamics, and agents' behaviors between the two environments. To address this issue, researchers have explored three main approaches: sim2real, digital twins (DTs), and parallel intelligence (PI) technologies. This paper reviews these solutions and examines their applications and innovations in autonomous driving. Furthermore, we delve into the state-of-the-art algorithms, models, and involved simulators, and discuss the developmental process from sim2real to DTs and PI. The presentation also sheds light on the challenges and future perspectives in the development of sim2real, DTs, and PI in the field of autonomous driving.

Federated learning (FL) provides a high efficient decentralized machine learning framework, where the training data remains distributed at remote clients in a network. Though FL enables a privacy-preserving mobile edge computing framework using IoT devices, recent studies have shown that this approach is susceptible to poisoning attacks from the side of remote clients. To address the poisoning attacks on FL, we provide a two-phase defense algorithm called Local Malicious Factor (LoMar). In phase I, LoMar scores model updates from each remote client by measuring the relative distribution over their neighbors using a kernel density estimation method. In phase II, an optimal threshold is approximated to distinguish malicious and clean updates from a statistical perspective. Comprehensive experiments on four real-world datasets have been conducted, and the experimental results show that our defense strategy can effectively protect the FL system. Specifically, the defense performance on Amazon dataset under a label-flipping attack indicates that, comparing with FG+Krum, LoMar increases the target label testing accuracy from $96.0\%$ to $98.8\%$ , and the overall averaged testing accuracy from 90.1% to $97.0%.

Motion planning is one of the most significant part in autonomous driving. Learning-based motion planning methods attract many researchers' attention due to the abilities of learning from the environment and directly making decisions from the perception. The deep Q-network, as a popular reinforcement learning method, has achieved great progress in autonomous driving, but these methods seldom use the global path information to handle the issue of directional planning such as making a turning at an intersection since the agent usually learns driving strategies only by the designed reward function, which is difficult to adapt to the driving scenarios of urban roads. Moreover, different motion commands such as the steering wheel and accelerator are associated with each other from classic Q-networks, which easily leads to an unstable prediction of the motion commands since they are independently controlled in a practical driving system. In this paper, a conditional deep Q-network for directional planning is proposed and applied in end-to-end autonomous driving, where the global path is used to guide the vehicle to drive from the origination to the destination. To handle the dependency of different motion commands in Q-networks, we take use of the idea of fuzzy control and develop a defuzzification method to improve the stability of predicting the values of different motion commands. We conduct comprehensive experiments in the CARLA simulator and compare our method with the state-of-the-art methods. Experimental results demonstrate the proposed method achieves better learning performance and driving stability performance than other methods.

Since the state-of-the-art deep learning algorithms demand a large training dataset, which is often unavailable in some domains, the transfer of knowledge from one domain to another has been a trending technique in the computer vision field. However, this method may not be a straight-forward task considering several issues such as original network size or large differences between the source and target domain. In this paper, we perform transfer learning for semantic segmentation of off-road driving environments using a pre-trained segmentation network called DeconvNet. We explore and verify two important aspects regarding transfer learning. First, since the original network size was very large and did not perform well for our application, we proposed a smaller network, which we call the light-weight network. This light-weight network is half the size to the original DeconvNet architecture. We transferred the knowledge from the pre-trained DeconvNet to our light-weight network and fine-tuned it. Second, we used synthetic datasets as the intermediate domain before training with the real-world off-road driving data. Fine-tuning the model trained with the synthetic dataset that simulates the off-road driving environment provides more accurate results for the segmentation of real-world off-road driving environments than transfer learning without using a synthetic dataset does, as long as the synthetic dataset is generated considering real-world variations. We also explore the issue whereby the use of a too simple and/or too random synthetic dataset results in negative transfer. We consider the Freiburg Forest dataset as a real-world off-road driving dataset.

In the context of autonomous driving, the existing semantic segmentation concept strongly supports on-road driving where hard inter-class boundaries are enforced and objects can be categorized based on their visible structures with high confidence. Due to the well-structured nature of typical on-road scenarios, current road extraction processes are largely successful and most types of vehicles are able to traverse through the area that is detected as road. However, the off-road driving domain has many additional uncertainties such as uneven terrain structure, positive and negative obstacles, ditches, quagmires, hidden objects, etc. making it very unstructured. Traversing through such unstructured area is constrained by a vehicle’s type and its capability. Therefore, an alternative approach to segmentation of the off-road driving trail is required that supports consideration of the vehicle type in a way that is not considered in state-of-the-art on-road segmentation approaches. To overcome this limitation and facilitate the path extraction in the off-road driving domain, we propose traversability concept and corresponding dataset which is based on the notion that the driving trails should be finely resolved into different sub-trails and areas corresponding to the capability of different vehicle classes in order to achieve safe traversal. Based on this, we consider three different classes of vehicles (sedan, pickup, and off-road) and label the images corresponding to the traversing capability of those vehicles. So the proposed dataset facilitates the segmentation of off-road driving trail into three regions based on the nature of the driving area and vehicle capability. We call this dataset as CaT (CAVS Traversability) dataset and is publicly available at https://www.cavs.msstate.edu/resources/downloads/CaT/CaT.tar.gz.