In this paper, we propose a novel variant of the algorithm to improve the generalization performance for Elman neural networks (ENN). Here, the weight decay term, also called $L_2$ regularization, which can effectively control the value of weights excessive growth, also over-fitting phenomenon can be effectively prevented. The main contribution of this work lies in that we have conducted a rigorous theoretical analysis of the proposed approach, i.e. the weak and strong convergence results are obtained. The comparison experiments to the problems of function approximation and classification on the real-world data have been performed to verify the theoretical results.

Extreme learning machine (ELM) is a simple feedforward neural network, and it has been extensively used in applications for its extremely fast learning speed and good generalization performance. Nevertheless, it is implemented normally under the empirical risk minimization scheme and the model trained by ELM is prone to overfitting. In addition, the ELM provides more nodes than it actually needs, this means that the network structure is not sparse enough. To solve above problems, two efficient algorithms for training ELM: Group L _1/2 regularization and smoothing Group L _1/2 regularization methods are proposed in this article. But, the basic group L _1/2 regularization is nondifferentiable at the origin and which causes oscillation. So, we modify the basic group L _1/2 regularization by smoothing it at the origin. Simulation results show that the ELM with smoothing group L _1/2 regularization can effectively prune redundant nodes and redundant weights of the surviving nodes, which has better performance than traditional ELM, the ELM with L ₁ regularization method, and with group L _1/2 regularization method.

This paper presents new theoretical results on the backpropagation algorithm with smoothing $$L_{1/2}$$ regularization and adaptive momentum for feedforward neural networks with a single hidden layer, i.e., we show that the gradient of error function goes to zero and the weight sequence goes to a fixed point as n (n is iteration steps) tends to infinity, respectively. Also, our results are more general since we do not require the error function to be quadratic or uniformly convex, and neuronal activation functions are relaxed. Moreover, compared with existed algorithms, our novel algorithm can get more sparse network structure, namely it forces weights to become smaller during the training and can eventually removed after the training, which means that it can simply the network structure and lower operation time. Finally, two numerical experiments are presented to show the characteristics of the main results in detail.

Recently there have been renewed interests in high order neural networks (HONNs) for its powerful mapping capability. Ridge polynomial neural network (RPNN) is an important kind of HONNs, which always occupies a key position as an efficient instrument in the tasks of classification or regression. In order to make the convergence speed faster and the network generalization ability stronger, we introduce a regularization model for RPNN with Group Lasso penalty, which deals with the structural sparse problem at the group level in this paper. Nevertheless, there are two main obstacles for introducing the Group Lasso penalty, one is numerical oscillation and the other is convergence analysis challenge. In doing so, we adopt smoothing function to approximate the Group Lasso penalty to overcome these drawbacks. Meanwhile, strong and weak convergence theorems, and monotonicity theorems are provided for this novel algorithm. We also demonstrate the efficiency of our proposed algorithm by numerical experiments, and compare it to the no regularizer, L2 regularizer, L1/2 regularizer, smoothing L1/2 regularizer, and the Group Lasso regularizer, and also the relevant theoretical analysis has been verified.