Deep neural networks often suffer from poor performance or even training failure due to the ill-conditioned problem, the vanishing/exploding gradient problem, and the saddle point problem. In this article, a novel method by acting the gradient activation function (GAF) on the gradient is proposed to handle these challenges. Intuitively, the GAF enlarges the tiny gradients and restricts the large gradient. Theoretically, this article gives conditions that the GAF needs to meet and, on this basis, proves that the GAF alleviates the problems mentioned above. In addition, this article proves that the convergence rate of SGD with the GAF is faster than that without the GAF under some assumptions. Furthermore, experiments on CIFAR, ImageNet, and PASCAL visual object classes confirm the GAF's effectiveness. The experimental results also demonstrate that the proposed method is able to be adopted in various deep neural networks to improve their performance. The source code is publicly available at https://github.com/LongJin-lab/Activated-Gradients-for-Deep-Neural-Networks.

In this technical note, to seek the optimal solution to time-dependent nonlinear optimization subject to linear inequality and equality constraints (TDNO-IEC), the gradient-based differential neural-solution, termed as GDN model, is proposed and researched. Notably, TDNO-IEC is firstly converted into the non-homogeneous linear equation with the dynamic parameter. Second, differential neural-solution with the aid of gradient is designed. The contrastive theoretical analyses among the GDN model, gradient-based neural network (GNN), and the dual neural network (DNN) prove that the proposed GDN model has higher accuracy for eliminating the large solution error with exponential convergence. In addition, reasonable convergent time of the GDN model is guaranteed by activation functions with simple formulation. Last, an illustrative example and real-world applications, including robot motion planning and data dimension reduction and reconstruction, further validate the high availability of the proposed GDN model.

For quadratic programming (QP), it is usually assumed that the solving process is free of measurement noises or that the denoising has been conducted before the computation. However, time is precious for time-varying QP (TVQP) in practice. Preprocessing for denoising may consume extra time, and consequently violates real-time requirements. Therefore, a model with inherent noise tolerance is urgently needed to solve TVQP problems in real time. In this paper, we make progress along this direction by proposing a modified Zhang neural network (MZNN) model for the solution of TVQP. The original Zhang neural network model and the gradient neural network model are employed for comparisons with the MZNN model. In addition, theoretical analyses show that, without measurement noise, the proposed MZNN model globally converges to the exact real-time solution of the TVQP problem in an exponential manner and that, in the presence of measurement noises, the proposed MZNN model has a satisfactory performance. Finally, two illustrative simulation examples as well as a physical experiment are provided and analyzed to substantiate the efficacy and superiority of the proposed MZNN model for TVQP problem solving.

Enlightened by competitive and collaborative coordination behaviors widely observed in natural swarm systems, this work emphasizes these coordinating modes in multirobot systems and optimizes system stability along with resource utilization. Then, schemes are constructed to describe and model these two modes, where a $k$ -winner-take-all concept is introduced as the driving principle of multirobot competition. In addition, a distributed coordination approach is established to effectively handle the above schemes aided with optimality theory, which is developed by a fusion of a recurrent neural dynamics solver and a distributed solver. The former is a single-layer neural dynamics model with a simple structure, and the latter transforms the involved global information to a distributed type via consensus. Both of them are carried out in the discrete-time domain to fit the actual application. Finally, the convergence and stability of the proposed coordination approach are proved via theoretical analysis and further demonstrated through simulations and experiments.

In this paper, a neural-dynamic distributed scheme is proposed for the cooperative control of multiple redundant manipulators with limited communications. It is guaranteed that, with the communication network being connected, all manipulators can jointly reach the same desired motion. The proposed distributed scheme is rearranged as a time-varying quadratic program and solved online by a Zhang neural network. Then, theoretical analyses show that, without noise, the proposed distributed scheme is able to execute a given task with exponentially convergent position errors. Moreover, an explicit bound relationship between the control input noise and the end-effector position error is analytically derived. Furthermore, numerical comparisons substantiate the superiority, effectiveness, and accuracy of the proposed distributed scheme.

Community detection is a popular yet thorny issue in social network analysis. A symmetric and nonnegative matrix factorization (SNMF) model based on a nonnegative multiplicative update (NMU) scheme is frequently adopted to address it. Current research mainly focuses on integrating additional information into it without considering the effects of a learning scheme. This study aims to implement highly accurate community detectors via the connections between an SNMF-based community detector’s detection accuracy and an NMU scheme’s scaling factor. The main idea is to adjust such scaling factor via a linear or nonlinear strategy, thereby innovatively implementing several scaling-factor-adjusted NMU schemes. They are applied to SNMF and graph-regularized SNMF models to achieve four novel SNMF-based community detectors. Theoretical studies indicate that with the proposed schemes and proper hyperparameter settings, each model can: 1) keep its loss function nonincreasing during its training process and 2) converge to a stationary point. Empirical studies on eight social networks show that they achieve significant accuracy gain in community detection over the state-of-the-art community detectors.