In this research, a new performance assessment based on the Probability of Detection (POD) reliability measure is developed integrating and discussing the effect of further parameters on classification results and therefore establishing a new connection between relevant process parameters and the related classifier evaluation. To illustrate the introduced approach, machine learning-based recognition of complex driving situations for human drivers is interpreted. Based on sensor signals and a complex driving scenario, related dynamical changes are classified and compared using the POD approach. Based on the POD-related evaluation, different machine learning approaches can be clearly distinguished with respect to their ability to predict the correct driver behavior as a function of time prior to the event itself. The introduced approach allows a very detailed comparison of classifiers related to the effects of parameters affecting the processes to be classified. In addition to recently published results on this novel approach, an extension of the POD approach by considering false positives and varying decision threshold in the comparison process is proposed. Generalization of the introduced approach for binary and continuous data is presented.

Structural health monitoring systems are based on suitable sensor techniques allowing online and offline supervision of technical systems. The quantification of sensors/measurement devices is a key issue for qualifying their effectiveness and efficiency and therefore to ensure safe operations. Probability of detection serves as a performance measure for quantifying the reliability of conventional nondestructive testing procedures taking into account statistical variability of sensor-based measurements. For vibration-based supervision approaches and fault detection and isolation methods, the probability of detection approach cannot be applied similarly. This results mainly from the complexity of the dynamical behavior of systems monitored in relation to faults, sensors position (observability), and the related feature extraction or monitoring task. In this contribution, probability of detection evaluation of vibration-based fault detection of elastic mechanical structures to be monitored is developed. Beside a principal discussion of the problem serving as introduction, an example using different sensor types in combination with mechanical modifications of an elastic beam is presented. The a_90/95-criteria representing probability of 90% detection at a confidence level of 95% is examined to the measurements and related outcomes. Based on the analysis of a suitably chosen feature (like eigenfrequency or band power) and dependent on the mechanical modes considered, the efficiency and deficiency of the different combinations are shown. Based on the proposed approach, a new insight into the usefulness of different sensor type and fault position combination becomes possible. To improve the detection quality, suitable assumptions in combination with sensor/information fusion are applied to feature-based analysis as detection task using vibration measurements. In addition, based on an experimental evaluation, it can be concluded that a suitable fault-feature probability of detection analysis can be successfully implemented as a new reliable measure for vibration-based fault detection and isolation approaches. Furthermore, decision fusion as the combination of different measurements will allow the improvement of results. Dependent on noise analysis, a trade-off between flaw size detection and probability of falsely characterizing a fault with a_90/95 reliability level can be attained.

Vision-based object detection plays a crucial role for the complete functionality of many engineering systems. Typically, detectors or classifiers are used to detect objects or to distinguish different targets. This contribution presents a new evaluation of CNN classifiers in image detection using a modified Probability of Detection reliability measure. The proposed method allows the evaluation of further image parameters affecting the classification results. The proposed evaluation method is implemented on images and comparisons made on parameters with the best detection capability. A typical certification standard ($90/95$) denoting a 90% probability of detection at 95% reliability level is adapted and successfully applied. Using the $90/95$ standard, comparisons are made between different image parameters. A noise analysis procedure is introduced, permitting the trade-off between the detection rate, false alarms, and process parameters. The advantage of the novel approach is experimentally evaluated for vision-based classification results of CNN considering different image parameters. With this new POD evaluation, classifiers will become a trustworthy part of vision systems.

Reliability assessment of Structural Health Monitoring systems applied to the diagnosis of faults in elastic structures is discussed in this contribution. In the field of Non Destructive Testing (NDT), Probability of Detection (POD) is used as a performance measure for quantifying the reliability of NDT approaches. However, reliability measures applied to Fault Detection and Isolation (FDI) is not discussed much in this research area. In this contribution the reliability of vibration-based monitoring approaches with respect to their principal ability to detect changes, realize diagnosis, and isolate causes (as FDI) is discussed. Using eigenfrequency and band power as attributes, a novel feature-based POD is proposed and implemented as a reliability measure for vibration-based FDI. The a_90/95 criteria which represents 90% probability of detecting a fault with 95% level of confidence is successfully implemented to an experimentally realized monitoring system. Emphases is made on improving the detection quality through sensor/information fusion. Acceleration, strain, and deflection measurement paths are utilized for diagnosis purposes based on experimental results.