The paper proposes a formal approach for describing and evaluating the datasets that are used in automotive applications for machine learning, testing, and validation purposes. Proper, that is, qualitative and quantitative characterization of the datasets can simplify the analysis, evaluation, and comparison of perception-based algorithms designed for highly automated vehicles. Such formalism is also needed to achieve compliance with the automotive industry safety standards that have been recently introduced. Characterization in the form of size or type of raw data, number of recognized and classified objects, and environmental parameters is not perfectly suitable for describing both the static and dynamic aspects of automotive datasets; therefore, another approach is required. In this paper, an efficient method based on an object tracking mechanism, grid representation of the sensor field of view, heatmap concept, and Wasserstein metric is proposed. The efficiency of the method is demonstrated by its ability to handle both the size, properties, and diversity of the dataset, including static and time-varying aspects. The presented description can also be used to compare different datasets and to define the amount of data to be collected.

The aim of this work is to formulate a new metric to be used in the automotive industry for the evaluation process of software used to detect vehicles on video data. To achieve this goal, we have formulated a new concept for measuring the degree of matching between rectangles for industrial use. We propose new measure based on three sub-measures focused on the area of the rectangle, its shape, and distance. These sub-measures are merged into a General similarity measure to avoid problems with poor adaptability of the Jaccard index to practical issues of recognition. Additionally, we create method of calculation of detection quality in the sequence of video frames that summarizes the local quality and adds information about possible late detection. Experiments with real and artificial data have confirmed that we have created flexible tools that can reduce time needed to evaluate detection software efficiently, and provide more detailed information about the quality of detection than the Jaccard index. Their use can significantly speed up data analysis and capture the weaknesses and limitations of the detection system under consideration. Our detection quality assessment method can be of interest to all engineers involved in machine recognition of video data.

An environmental perception aims to detect and classify objects existing in the vehicle's surroundings to build a virtual representation of the world. It can be considered as an essential building block in modern advanced safety and automated driving systems. These systems implement functionalities that are very often classified as safety-critical. Testing is actually the only method to assure the required system quality and compliance with safety standards. Testing the safety aspects of automated driving systems creates a number of challenges related to the adequacy and completeness of the test cases being developed. This is because the number of possible test scenarios on the road is actually infinite and such scenarios are time- and space-dependent. In this paper, we define a qualitative and quantitative measure that can be used to evaluate the completeness and adequacy of the test scenarios and consequently limit the amount of tests needed. The measure allows to consider static aspects of the perception systems such as detection and classification, as well as dynamic aspects related to object tracking.

While machine learning models are powering more and more everyday devices, there is a growing need for explaining them. This especially applies to the use of deep reinforcement learning in solutions that require security, such as vehicle motion planning. In this paper, we propose a method for understanding what the RL agent’s decision is based on. The method relies on conducting a statistical analysis on a massive set of state-decisions samples. It indicates which input features have an impact on the agent’s decision and the relationships between the decisions, the significance of the input features, and their values. The method allows us to determine whether the process of making a decision by the agent is coherent with human intuition and what contradicts it. We applied the proposed method to the RL motion planning agent which is supposed to drive a vehicle safely and efficiently on a highway. We find out that making such an analysis allows for a better understanding of the agent’s decisions, inspecting its behavior, debugging the ANN model, and verifying the correctness of the input values, which increases its credibility.

While machine learning models are powering more and more everyday devices, there is a growing need for explaining them. This especially applies to the use of Deep Reinforcement Learning in solutions that require security, such as vehicle motion planning. In this paper, we propose a method of understanding what the RL agent’s decision is based on. The method relies on conducting statistical analysis on a massive set of state-decisions samples. It indicates which input features have an impact on the agent’s decision and the relationships between decisions, the significance of the input features, and their values. The method allows us for determining whether the process of making a decision by the agent is coherent with human intuition and what contradicts it. We applied the proposed method to the RL motion planning agent which is supposed to drive a vehicle safely and efficiently on a highway. We find out that making such analysis allows for a better understanding agent’s decisions, inspecting its behavior, debugging the ANN model, and verifying the correctness of input values, which increases its credibility.

The goal of this paper is to present results of analysis of artificially generated disturbances imitating real defects of camera that occurs in the process of testing autonomous vehicles both during rides and later, in vehicle software simulation and their impact on quality of object detection. We focus on one perception module responsible for detection of other moving vehicles on the road. Injected faults are obliteration by particles and pulsating blinding. At the same time, we want to propose an examination approach scheme that will provide detailed information about distribution of quality in this comparative experiment. The method can be reused for different perception modules, faults, scene sets and also in order to compare new releases of main recognition software. To do so we combine statistical methods (Welch's ANOVA) and topological analysis (Clusterization over space of distributions, Wasserstein metric). Work provides a summary of the experiment for all data used and described by mentioned tools and examples of certain cases that illustrate general conclusions.