For environmental perception, autonomous vehicles and intelligent roadside infrastructure systems contain multiple sensors, i.e. radar (Radio Detection and Ranging), lidar (Light Detection and Ranging), and camera sensors with the aim to detect, classify and track multiple road users. Data from multiple sensors are fused together to enhance the perception quality of the sensor system because each sensor has strengths and weaknesses, e.g. resolution, distance measurement, and dependency on weather conditions. For data fusion, it is necessary to transform the data from the different sensor coordinates to a common coordinate frame. This process is referred to as multi-sensor calibration and is a challenging task, which is mostly performed manually. This paper introduces a new method for auto-calibrating three-dimensional (3D) radar, 3D lidar, and red-green-blue (RGB) mono-camera sensors using a static multi-target-based system. The proposed method can be used with sensors operating at different frame rates without time synchronization. Furthermore, the described static multi-target system is cost-effective, easy to build, and applicable for short to long-distance calibration. The experimental results for multiple sets of measurements show good results with projection errors measured as maximum root mean square error (RMSE) of (u,v) = (2.4, 1.8) pixels for lidar-to-camera calibration, RMSE of (u,v) = (2.2, 3.0) pixels for 3D radar-to-camera calibration and RMSE of (x,y,z) = (2.6, 2.7, 14.0) centimeters for 3D radar-to-lidar calibration.

Mono RGB cameras and automotive radar sensors provide a complementary information set that makes them excellent candidates for sensor data fusion to obtain robust traffic user detection. This has been widely used in the vehicle domain and recently introduced in roadside-mounted smart infrastructure-based road user detection. However, the performance of the most commonly used late fusion methods often degrades when the camera fails to detect road users in adverse environmental conditions. The solution is to fuse the data using deep neural networks at the early stage of the fusion pipeline to use the complete data provided by both sensors. Research has been carried out in this area, but is limited to vehicle-based sensor setups. Hence, this work proposes a novel deep neural network to jointly fuse RGB mono-camera images and 3D automotive radar point cloud data to obtain enhanced traffic user detection for the roadside-mounted smart infrastructure setup. Projected radar points are first used to generate anchors in image regions with a high likelihood of road users, including areas not visible to the camera. These anchors guide the prediction of 2D bounding boxes, object categories, and confidence scores. Valid detections are then used to segment radar points by instance, and the results are post-processed to produce final road user detections in the ground plane. The trained model is evaluated for different light and weather conditions using ground truth data from a lidar sensor. It provides a precision of 92%, recall of 78%, and F1-score of 85%. The proposed deep fusion methodology has 33%, 6%, and 21% absolute improvement in precision, recall, and F1-score, respectively, compared to object-level spatial fusion output.

Environment perception using camera, radar, and/or lidar sensors has significantly improved in the last few years because of deep learning-based methods. However, a large group of these methods fall into the category of supervised learning, which requires a considerable amount of annotated data. Due to uncertainties in multi-sensor data, automating the data labeling process is extremely challenging; hence, it is performed manually to a large extent. Even though full automation of such a process is difficult, semi-automation can be a significant step to ease this process. However, the available work in this regard is still very limited; hence, in this paper, a novel semi-automatic annotation methodology is developed for labeling RGB camera images and 3D automotive radar point cloud data using a smart infrastructure-based sensor setup. This paper also describes a new method for 3D radar background subtraction to remove clutter and a new object category, GROUP, for radar-based object detection for closely located vulnerable road users. To validate the work, a dataset named INFRA-3DRC is created using this methodology, where 75% of the labels are automatically generated. In addition, a radar cluster classifier and an image classifier are developed, trained, and tested on this dataset, achieving accuracy of 98.26% and 94.86%, respectively. The dataset and Python scripts are available at https://fraunhoferivi.github.io/INFRA-3DRC-Dataset/.

Advanced sensors like Radar, LiDAR and camera are used in development for autonomous vehicles for environment perception to do real time object detection, classification and tracking of dynamic objects. The process of environment perception is a crucial step in development of autonomous vehicles but it is unfortunately, one of the most challenging processes because the outside environment is unpredictable, continuously changing and full of uncertainties. In this situation, the maximum assistance to such vehicles is of utmost need. Development of intelligent infrastructure unit, known as road ride unit (RSU), which can be mounted at various specific locations on the road sides to perceive environment in real time from high angle view can assist automated driving. Such RSUs can consist of various types and combination of sensors and vehicle to infrastructure (V21) communication unit. It can send the information about critical situations, traffic alerts and other important information to nearby vehicles in real time to further enhance automated driving. This paper describes the details of development of intelligent infrastructure for a test field within the research project IN2Lab. test field development. It describes the architecture of RSUs and the concept of environment perception including sensor requirement and selection, sensor field of view, sensor coordinate system and concept of sensor fusion.

Smart cities rely on intelligent infrastructure to enhance road safety, optimize traffic flow, and enable vehicle-to-infrastructure (V2I) communication. A key component of such infrastructure is an efficient and real-time perception system that accurately detects diverse traffic participants. Among various sensing modalities, automotive radar is one of the best choices due to its robust performance in adverse weather and low-light conditions. However, due to low spatial resolution, traditional clustering-based approaches for radar object detection often struggle with vulnerable road user detection and nearby object separation. Hence, this paper proposes a deep learning-based $$3+1$$ D radar point cloud clustering methodology tailored for smart infrastructure-based perception applications. This approach first performs semantic segmentation of the radar point cloud, followed by instance segmentation to generate well-formed clusters with class labels using a deep neural network. It also detects single-point objects that conventional methods often miss. The described approach is developed and experimented using a smart infrastructure-based sensor setup and it performs segmentation of the point cloud in real-time. Experimental results demonstrate 95.35% F1-macro score for semantic segmentation and 91.03% mean average precision (mAP) at an intersection over union (IoU) threshold of 0.5 for instance segmentation. Further, the complete pipeline operates at 43.61 frames per second with a memory requirement of less than 0.7 MB on the edge device (Nvidia Jetson AGX Orin). We will release the RoadsideRadar dataset along with the code implementation of this work at https://github.com/bhanderisavan/roadside-radar-seg .