The majority of existing neural networks operate with real-valued representation of data. However, there are multiple tasks in which the input is complex-valued. The complex-valued data is considered to be more informative in terms of larger representational capacity. These reasons motivate researchers to develop neural networks using complex numbers instead of real-valued ones. In this paper, we take a step forward in the generalization of neural networks. We develop the basic building blocks for dual-valued neural networks based on dual numbers. We adjust basic layers such as Linear, Convolution, Average Pooling, ReLU to the dual domain and present an algorithm for Dual Batch Normalization. We construct several dual-valued neural networks for classification tasks basing on classical CV problems and the MusicNet and G2Net datasets. We show that dual-valued models outperform analogous complex-valued neural networks in execution time and have higher or at least the same accuracy.

Due to industrial demands to handle increasing amounts of training data, lower the cost of computing one model at a time, and lessen the ecological effects of intensive computing resource consumption, the job of speeding the training of deep neural networks becomes exceedingly challenging. Adaptive Online Importance Sampling and IDS are two brand-new methods for accelerating training that are presented in this research. On the one hand, Adaptive Online Importance Sampling accelerates neural network training by lowering the number of forward and backward steps depending on how poorly a model can identify a given data sample. On the other hand, Intellectual Data Selection accelerates training by removing semantic redundancies from the training dataset and subsequently lowering the number of training steps. The study reports average 1.9x training acceleration for ResNet50, ResNet18, MobileNet v2 and YOLO v5 on a variety of datasets: CIFAR-100, CIFAR-10, ImageNet 2012 and MS COCO 2017, where training data are reduced by up to five times. Application of Adaptive Online Importance Sampling to ResNet50 training on ImageNet 2012 results in 2.37 times quicker convergence to 71.7% top-1 accuracy, which is within 5% of the baseline. Total training time for the same number of epochs as the baseline is reduced by 1.82 times, with an accuracy drop of 2.45 p.p. The amount of time required to apply Intellectual Data Selection to ResNet50 training on ImageNet 2012 is decreased by 1.27 times with a corresponding decline in accuracy of 1.12 p.p. Applying both methods to ResNet50 training on ImageNet 2012 results in 2.31 speedup with an accuracy drop of 3.5 p.p.

Artificial neural networks have become an inseparable element of human life. Researches do not stop at the current progress and try to improve neural networks and expand fields of applications. The most widespread way to make models better consists in generalization of existing methods and approaches. In this paper, we make a step in an unusual direction: we propose to use neural networks based on dual numbers. We develop a special subclass of dual-valued operators, which satisfy the equivalent of the Cauchy-Riemann equations for the dual domain. We also propose a new type of preprocessing and batch normalization, relying on peculiarities of dual numbers. We test deep holomorphic dual-valued models on music transcription and gravitational wave detection tasks and show that our holomorphic dual-valued networks achieve better inference time compared to the dual-valued models and are better than their real-valued counter-parts in sense of metrics.

The vast majority of existing neural networks operate by rules set within the algebra of real numbers. However, as theoretical understanding of the fundamentals of neural networks and their practical applications grow stronger, new problems arise, which require going beyond such algebra. Various tasks come to light when the original data naturally have complex-valued formats. This situation is encouraging researchers to explore whether neural networks based on complex numbers can provide benefits over the ones limited to real numbers. Multiple recent works have been dedicated to developing the architecture and building blocks of complex-valued neural networks. In this paper, we generalize models by considering other types of hypercomplex numbers of the second order: dual and double numbers. We developed basic operators for these algebras, such as convolution, activation functions, and batch normalization, and rebuilt several real-valued networks to use them with these new algebras. We developed a general methodology for dual and double-valued gradient calculations based on Wirtinger derivatives for complex-valued functions. For classical computer vision (CIFAR-10, CIFAR-100, SVHN) and signal processing (G2Net, MusicNet) classification problems, our benchmarks show that the transition to the hypercomplex domain can be helpful in reaching higher values of metrics, compared to the original real-valued models.