We demonstrate a working functional prototype of a cooperative perception system that maintains a real-time digital twin of the traffic environment, providing a more accurate and more reliable model than any of the participant subsystems—in this case, smart vehicles and infrastructure stations—would manage individually. The importance of such technology is that it can facilitate a spectrum of new derivative services, including cloud-assisted and cloud-controlled ADAS functions, dynamic map generation with analytics for traffic control and road infrastructure monitoring, a digital framework for operating vehicle testing grounds, logistics facilities, etc. In this paper, we constrain our discussion on the viability of the core concept and implement a system that provides a single service: the live visualization of our digital twin in a 3D simulation, which instantly and reliably matches the state of the real-world environment and showcases the advantages of real-time fusion of sensory data from various traffic participants. We envision this prototype system as part of a larger network of local information processing and integration nodes, i.e., the logically centralized digital twin is maintained in a physically distributed edge cloud.

A spectacular measurement campaign was carried out on a real-world motorway stretch of Hungary with the participation of international industrial and academic partners. The measurement resulted in vehicle based and infrastructure based sensor data that will be extremely useful for future automotive R&D activities due to the available ground truth for static and dynamic content. The aim of the measurement campaign was twofold. On the one hand, road geometry was mapped with high precision in order to build Ultra High Definition (UHD) map of the test road. On the other hand, the vehicles—equipped with differential Global Navigation Satellite Systems (GNSS) for ground truth localization—carried out special test scenarios while collecting detailed data using different sensors. All of the test runs were recorded by both vehicles and infrastructure. The paper also showcases application examples to demonstrate the viability of the collected data having access to the ground truth labeling. This data set may support a large variety of solutions, for the test and validation of different kinds of approaches and techniques. As a complementary task, the available 5G network was monitored and tested under different radio conditions to investigate the latency results for different measurement scenarios. A part of the measured data has been shared openly, such that interested automotive and academic parties may use it for their own purposes.

Advanced driving assistance systems (ADASs) are critical for automotive safety. They rely on various sensors (especially with an increasing reliance on visual sensors to meet evolving safety standards) to capture relevant environmental data. The validation of ADAS systems is crucial to ensure their reliability and performance in real-world driving scenarios; however, this requires reference data. This paper focuses on the development of a reference sensor system that can provide reference data and does support the validation of visual sensors for ADAS systems. The system is validated in various relevant scenarios at an automotive proving ground.

The representation of objects in LiDAR point clouds is changed as the height of the mounting position of sensor devices gets increased. Most of the available open datasets for training machine learning based object detectors are generated with vehicle top mounted sensors, thus the detectors trained on such datasets perform weaker when the sensor is observing the scene from a significantly higher viewpoint (e.g. infrastructure sensor). In this paper a novel Automatic Label Injection method is proposed to label the objects in the point cloud of the high-mounted infrastructure LiDAR sensor based on the output of a well performing “trainer” detector deployed at optimal height while considering the uncertainties caused by various factors described in detail throughout the paper. The proposed automatic labeling approach has been validated on a small scale sensor setup in a real-world traffic scenario where accurate differential GNSS reference data where also available for each test vehicle. Furthermore, the concept of a distributed multi-sensor system covering a larger area aimed for automatic dataset generation is also presented. It is shown that a machine learning based detector trained on differential GNSS-based training dataset performs very similarly to the detector retrained on a dataset generated by the proposed Automatic Label Injection technique. According to our results a significant increase in the maximum detection range can be achieved by retraining the detector on viewpoint specific data generated fully automatically by the proposed label injection technique compared to a detector trained on vehicle top mounted sensor data.

Using point cloud sequences is a popular way to harness the additional information represented in the time domain in order to enhance the performance of 3D object detector neural networks. However, it is not trivial to decide which abstraction level should the additional information presented to the network, or what is the point in the architecture, where aggregating the additional information is most beneficial. In this article, the authors propose various voxel-based networks and analyze their performance in relation to the abstraction level of the time series data. During the evaluation, the authors examine the object detection performance of a popular voxel-based neural network with its original architecture and several variants where the time domain related features were propagated through the network and aggregated at different stages of processing. Based on the evaluation results, a conclusion is drawn regarding the abstraction level at which the time-series aggregation step is performed in order to improve the performance of the baseline voxel-based detector.