31 January 2022 in Journal of Signal Processing Systems
Testing and evaluation of an automotive perception system is a complicated task which requires special equipment and infrastructure. To compute key performance indicators and compare the results with real-world situation, some additional sensors and manual data labelling are often required. In this article, we propose a different approach, which is based on a UAV equipped with a 4K camera flying above a test track. Two computer vision methods are used to precisely determine the positions of the objects around the car – one based on ArUco markers and the other on a DCNN (we provide the algorithms used on GitHub). The detections are then correlated with the perception system readings. For the static and dynamic experiments, the differences between various systems are mostly below 0.5 m. The results of the experiments performed indicate that this approach could be an interesting alternative to existing evaluation solutions.
20 January 2022 in IEEE Access
The paper proposes a formal approach for describing and evaluating the datasets that are used in automotive applications for machine learning, testing, and validation purposes. Proper, that is, qualitative and quantitative characterization of the datasets can simplify the analysis, evaluation, and comparison of perception-based algorithms designed for highly automated vehicles. Such formalism is also needed to achieve compliance with the automotive industry safety standards that have been recently introduced. Characterization in the form of size or type of raw data, number of recognized and classified objects, and environmental parameters is not perfectly suitable for describing both the static and dynamic aspects of automotive datasets; therefore, another approach is required. In this paper, an efficient method based on an object tracking mechanism, grid representation of the sensor field of view, heatmap concept, and Wasserstein metric is proposed. The efficiency of the method is demonstrated by its ability to handle both the size, properties, and diversity of the dataset, including static and time-varying aspects. The presented description can also be used to compare different datasets and to define the amount of data to be collected.
17 March 2023 in IEEE Access
The higher the vehicle autonomy level is, the more the problems that need to be solved by engineers designing its sensors and control systems. Apart from the mechanical, electrical, algorithmic and validation challenges, such projects require to efficiently gather, store and handle the very large amounts of data logged from the vehicles. This is because the key functionalities of the control system in automated vehicles are based on data that is collected from a variety of sources, processed and analyzed to generate actuation signals. The article goes through the challenges, opportunities as well as solutions associated with data logging, collection, storage, annotation, reprocessing and evaluation. It highlights that effective and efficient data handling is an essential element for obtaining the required performance, reliability, safety and quality capabilities that would allow the mass production of automated vehicles. In terms of the safety of passengers and other road users, the behavior of a mass-produced automated vehicle must be predictable and more reliable than that of an ordinary driver. The multitude of variants that test sequences must cover is practically endless. Therefore, shortening the time-to-market to an acceptable length is only possible if you use the right methods of working with data.
9 February 2022 in IEEE Access
Automotive radar sensors are vital in Advanced Driver Assistance Systems (ADAS). To be more precise, their ability to explicitly measure the relative velocity to its targets is essential in Adaptive Cruise Control (ACC) and Emergency Braking (EB) applications. Nevertheless, ADAS systems are getting more and more complex, due to constantly increasing demands regarding safety and performance. As a result, to speed up the development and validation time of ADAS systems, part of the testing is performed in simulations. Replacing some of the test drives by the runs in virtual environments not only reduces the cost of a product, but also helps in fully safe execution of dangerous corner cases. However, to enable reliable testing of radar-based ADAS systems in virtual environments, high-fidelity radar sensor models are required. In order to prove the reliability of a given model, a proper evaluation process has to be conducted. This paper presents an end-to-end, straightforward methodology for performance assessment and fine-tuning of radar sensor models. To show how the full pipeline of the framework can be executed, an exemplary radar sensor model has been incorporated. The successful fine-tuning of the model proves the usefulness of the introduced method.