In this paper, we propose a new first-order gradient-based algorithm to train deep neural networks. We first introduce the sign operation of stochastic gradients (as in sign-based methods, e.g., SIGN-SGD) into ADAM, which is called as signADAM. Moreover, in order to make the rate of fitting each feature closer, we define a confidence function to distinguish different components of gradients and apply it to our algorithm. It can generate more sparse gradients than existing algorithms do. We call this new algorithm signADAM++. In particular, both our algorithms are easy to implement and can speed up training of various deep neural networks. The motivation of signADAM++ is preferably learning features from the most different samples by updating large and useful gradients regardless of useless information in stochastic gradients. We also establish theoretical convergence guarantees for our algorithms. Empirical results on various datasets and models show that our algorithms yield much better performance than many state-of-the-art algorithms including SIGN-SGD, SIGNUM and ADAM. We also analyze the performance from multiple perspectives including the loss landscape and develop an adaptive method to further improve generalization. The source code is available at https://github.com/DongWanginxdu/signADAM-Learn-by-Confidence.
Deep learning has been demonstrated to be a universal and successful tool for classification, detection and segmentation. Convolutional neural network (CNN), as a representative model, achieves excellent performance in many image process areas due to its special structural design---shared weights, local receptive field and pooling layer. Large efforts are devoted to studying and improving model performance based on these three parts. In this paper, we proposed a novel pooling layer named restricted stochastic pooling layer. Restricted stochastic pooling randomly select activation from the top n maximum of each pooling region instead of the largest one (max pooling), or stochastic one (stochastic pooling). Our proposed method can not only obtain representative activation but also add randomness to the model. Through large amounts of experiments, we achieve good performance on SVHN and CIFAR-10 which demonstrates the effectiveness of restricted stochastic pooling.
Fast and accurate measuring and assessing phenotypic traits of soybean leaf play a key role in the soybean breeding. The manual analysis of soybean leaf is time-consuming and prone to human errors. In this paper, an automatic quantitative approach is proposed which can obtain the vital geometric parameters (e.g., major axis, minor axis, and tip angle) of soybean leaf based on key point detection model. Specifically, the proposed method utilizes the Stacked Hourglass Network to find and accurate localize the four key points of each soybean leaf. Once those interest points are identified, then the distances between them and the tip angle are computed automatically. By performing such strategy, the quantitative analysis of soybean leaf can be handled effectively. The extensive experimental results obtained on the collected soybean leaf dataset verify that the proposed method has better convergence accuracy and robustness than other classical methods.