A robust visual understanding of complex urban environments using passive optical sensors is an onerous and essential task for autonomous navigation. The problem is heavily characterized by the quality of the available dataset and the number of instances it includes. Regardless of the benchmark results of perception algorithms, a model would only be reliable and capable of enhanced decision making if the dataset covers the exact domain of the end-use case. For this purpose, in order to improve the level of instances in datasets used for the training and validation of Autonomous Vehicles (AV), Advanced Driver Assistance Systems (ADAS), and autonomous driving, and to reduce the void due to the no-existence of any datasets in the context of railway smart mobility, we introduce our multimodal hybrid dataset called ESRORAD. ESRORAD is comprised of 34 videos, 2.7 k virtual images, and 100 k real images for both road and railway scenes collected in two Normandy towns, Rouen and Le Havre. All the images are annotated with 3D bounding boxes showing at least three different classes of persons, cars, and bicycles. Crucially, our dataset is the first of its kind with uncompromised efforts on being the best in terms of large volume, abundance in annotation, and diversity in scenes. Our escorting study provides an in-depth analysis of the dataset’s characteristics as well as a performance evaluation with various state-of-the-art models trained under other popular datasets, namely, KITTI and NUScenes. Some examples of image annotations and the prediction results of our 3D object detection lightweight algorithms are available in ESRORAD dataset. Finally, the dataset is available online. This repository consists of 52 datasets with their respective annotations performed.

This paper presents a novel WiFi-Visual data fusion method for indoor robot (TIAGO++) localization. Long-term follow-up experiments show that this method can use 10 WiFi samples and 4 low-resolution images (58 × 58 in pixels) to localize an indoor robot with an average error distance of about 1.32 meters 3 months (or 1.7 meters 7 months ) after training data collection. Instead of neural network design, this paper focuses on soft data fusion to prevent unbounded errors in visual localization. The proposed soft data fusion includes first-layer WiFi-Visual feature fusion and second-layer decision vector fusion. Firstly, motivated by the excellent capability of the neural network in image processing and recognition, temporal-spatial features are extracted from WiFi data and represented in image form. Secondly, the image-form WiFi features and the visual features taken by the robot camera are combined together, and jointly exploited by a classification neural network to produce a likelihood vector for WiFi-Visual localization. This is called first-layer fusion. Similarly, these two types of features can be separately exploited by neural networks to produce another two independent likelihood vectors. Thirdly, the three likelihood vectors are fused by Hadamard product to produce a final likelihood vector. This is called second-layer fusion. The proposed soft data fusion does not apply any threshold or prioritize any data source over the other in the fusion process. It never excludes low-probability candidate positions, which can avoid information loss due to a hard decision. The demo video is provided^a, and the code will be open to the public after the publication of this work.