1 December 2020 in Machine Learning: Science and Technology
This paper introduces a novel algorithmic framework for a deep neural network (DNN), which in a mathematically rigorous manner, allows us to incorporate history (or memory) into the network—it ensures all layers are connected to one another. This DNN, called Fractional-DNN, can be viewed as a time-discretization of a fractional in time non-linear ordinary differential equation (ODE). The learning problem then is a minimization problem subject to that fractional ODE as constraints. We emphasize that an analogy between the existing DNN and ODEs, with standard time derivative, is well-known by now. The focus of our work is the Fractional-DNN. Using the Lagrangian approach, we provide a derivation of the backward propagation and the design equations. We test our network on several datasets for classification problems. Fractional-DNN offers various advantages over the existing DNN. The key benefits are a significant improvement to the vanishing gradient issue due to the memory effect, and better handling of nonsmooth data due to the network’s ability to approximate non-smooth functions.
18 May 2020 in Journal of Optimization Theory and Applications
10 December 2019 in arXiv
Recently the authors have studied a state and control constrained optimal control problem with fractional elliptic PDE as constraints. The goal of this paper is to continue that program forward and introduce an algorithm to solve such optimal control problems. We shall employ the well-known Moreau-Yosida regularization to handle the state constraints. Similarly to the classical case, we establish the convergence (with rate) of the regularized control problem to the original one. We discretize the problem using a finite element method and establish convergence of our numerical scheme. We emphasize that due to the non-smooth nature of the fractional Laplacian, the proof for the classical case does not apply to the fractional case. The numerical experiments confirm all our theoretical findings.