18 June 2021 in Analysis and data processing systems
The paper considers the problem of building robust neural networks using different robust loss functions. Applying such neural networks is reasonably when working with noisy data, and it can serve as an alternative to data preprocessing and to making neural network architecture more complex. In order to work adequately, the error back-propagation algorithm requires a loss function to be continuously or two-times differentiable. According to this requirement, two five robust loss functions were chosen (Andrews, Welsch, Huber, Ramsey and Fair). Using the above-mentioned functions in the error back-propagation algorithm instead of the quadratic one allows obtaining an entirely new class of neural networks. For investigating the properties of the built networks a number of computational experiments were carried out. Different values of outliers’ fraction and various numbers of epochs were considered. The first step included adjusting the obtained neural networks, which lead to choosing such values of internal loss function parameters that resulted in achieving the highest accuracy of a neural network. To determine the ranges of parameter values, a preliminary study was pursued. The results of the first stage allowed giving recommendations on choosing the best parameter values for each of the loss functions under study. The second stage dealt with comparing the investigated robust networks with each other and with the classical one. The analysis of the results shows that using the robust technique leads to a significant increase in neural network accuracy and in a learning rate.
18 December 2020 in Transaction of Scientific Papers of the Novosibirsk State Technical University
The paper is devoted to analyzing the ability of using robust functions for building neural networks. The research highlights different robust functions in terms of applying them for obtaining a robust modification of the back-propagation algorithm. The algorithm requires that the used loss function should be infinitely or continuously differentiable. The analysis of twelve different functions has been done. The derivate of Charbonnier function has been obtained by the author. The results of analysis shows which functions can be used for the further investigation and which ones should be excluded from it.