Understanding the surrounding environment of the vehicle is still one of the challenges for autonomous driving. This paper addresses 360-degree road scene semantic segmentation using surround view cameras, which are widely equipped in existing production cars. First, in order to address large distortion problem in the fisheye images, Restricted Deformable Convolution (RDC) is proposed for semantic segmentation, which can effectively model geometric transformations by learning the shapes of convolutional filters conditioned on the input feature map. Second, in order to obtain a large-scale training set of surround view images, a novel method called zoom augmentation is proposed to transform conventional images to fisheye images. Finally, an RDC based semantic segmentation model is built; the model is trained for real-world surround view images through a multi-task learning architecture by combining real-world images with transformed images. Experiments demonstrate the effectiveness of the RDC to handle images with large distortions, and that the proposed approach shows a good performance using surround view cameras with the help of the transformed images.

Event-based cameras have several advantages over traditional cameras that shoot videos in frames. Event cameras have a high temporal resolution, high dynamic range, and almost non-existence of blurriness. The data that is produced by event sensors forms a chain of events when a change in brightness is reported in each pixel. This feature makes it difficult to directly apply existing algorithms and take advantage of the event camera data. Due to the developments in neural networks, important advances were made in event-based image reconstruction. Even though these neural networks achieve precise reconstructions while preserving most of the properties of the event cameras, there is still an initialization time that needs to have the highest possible quality in the reconstructed frames. In this work, we present the SPADE-E2VID neural network model that improves the quality of early frames in an event-based reconstructed video, as well as the overall contrast. The SPADE-E2VID model improves the quality of the first reconstructed frames by 15.87% for MSE error, 4.15% for SSIM, and 2.5% in LPIPS. In addition, the SPADE layer in our model allows training our model to reconstruct videos without a temporal loss function. Another advantage of our model is that it has a faster training time. In a many-to-one training style, we avoid running the loss function at each step, executing the loss function at the end of each loop only once. In the present work, we also carried out experiments with event cameras that do not have polarity data. Our model produces quality video reconstructions with non-polarity events in HD resolution (1200 × 800). The Video, the code, and the datasets will be available at: https://github.com/RodrigoGantier/SPADE_E2VID.

Estimating when pedestrians cross the street is essential for intelligent transportation systems. Accurate, real-time prediction is critical to ensure the safety of the most vulnerable road users while improving passenger comfort. In the present work, we developed a model called Pedestrian Graph +, an improvement of our previous work, Pedestrian Graph, which predicts pedestrian crossing action in urban areas based on a Graph Convolution Network. We integrated two convolutional modules in the new model that provide additional context information (cropped images, cropped segmentation maps, ego-vehicle velocity data) to the main Graph Convolutional module, thus increasing accuracy. Our model is faster and smaller than other state-of-the-art models, achieving equivalent accuracy. Our model is faster than state-of-the-art models, with an inference time of 6 ms (on a GTX 1080) and low memory consumption (0.3 MB). We tested our model on two datasets, Joint Attention in Autonomous Driving (JAAD) and Pedestrian Intention Estimation (PIE), achieving 86% and 89% accuracy, respectively. Another contribution of our work is the ability to dynamically process almost any input size in the time domain without significant loss of accuracy. It is possible due to the fully convolutional property of ConvNets. Our models and results are available at https://github.com/RodrigoGantier/Pedestrian_graph_plus.

Simulators play a crucial role in autonomous driving, offering significant time, cost, and labor savings. Over the past few years, the number of simulators for autonomous driving has grown substantially. However, there is a growing concern about the validity of algorithms developed and evaluated in simulators, indicating a need for a thorough analysis of the development status of the simulators. To address existing gaps in research, this paper undertakes a comprehensive review of the history of simulators, proposes a utility-based taxonomy, and investigates the critical issues within open-source simulators. Analysis of the past thirty years' development trajectory reveals a trend characterized by an increase in open-source simulators and an expansion of their functionality scope. The categorization of simulators based on feature functionalities delineates five primary classes: traffic flow, sensory data, driving policy, vehicle dynamics, and comprehensive simulators. Furthermore, the paper identifies critical unresolved issues in open-source simulators, including concerns regarding the fidelity of sensory data, representation of traffic scenarios, and accuracy in vehicle dynamics simulation, all of which have the potential to undermine experimental confidence. Additionally, challenges in data format inconsistency, labor-intensive map construction processes, sluggish step updating, and insufficient support for Hardware-In-the-Loop testing are discussed as hindrances to experimental efficiency. In light of these findings, the survey furnishes task-oriented recommendations to aid in the selection of simulators, taking into account factors such as accessibility, maintenance status, and quality, while highlighting the inherent limitations of existing open-source simulators in validating algorithms and facilitating real-world experimentation.

The generalization performance for semantic segmentation remains a major challenge when the data distributions between the source and target domain mismatch. Unsupervised domain adaptation (UDA) approaches are proposed to mitigate the problem above, among which entropy-minimization-based methods have gained more and more attention. However, the methods merely follow the cluster assumption sharpening the prediction distribution, thus have limited performance improvement. Without additional priors, the entropy loss can easily over-sharpen the prediction distribution, which brings noisy information into the learning process. On the other hand, the gradient of the entropy loss is strongly biased toward easy samples, also leading to limited generalization advances. In this paper, we firstly propose a pixel-level consistency regularization method, which introduces the smoothness prior to the UDA problem. Furthermore, we propose the neutral cross-entropy loss based on the consistency regularization, and reveal that its internal neutralization mechanism mitigates the over-sharpness of entropy minimization via the flatness effect of consistency regularization. We also demonstrate that the gradient bias toward easy samples is inherently tackled via the neutral cross-entropy loss. The experiments show that the proposed method has outperformed state-of-the-art methods in two synthetic-to-real experiments, only using the lightweight network.

This letter presents a framework for 3D instance segmentation on point clouds. A 3D convolutional neural network is used as the backbone to generate semantic predictions and instance embeddings simultaneously. In addition to the embedding information, point clouds also provide 3D geometric information which reflects the relation between points. Considering both types of information, the structure-aware loss function is proposed to achieve discriminative embeddings for each 3D instance. To eliminate the quantization error caused by 3D voxel, the attention-based $k$ -nearest neighbor (kNN) is proposed. Different from the average strategy, it learns different weights for different neighbors to aggregate and update the instance embeddings. Our network can be trained in an end-to-end style. Experiments show that our approach achieves state-of-the-art performance on two challenging datasets for instance segmentation.