Communications in Computational Physics (CiCP) publishes original research and survey papers of high scientific value in computational modeling of physical problems. Results in multi-physics and multi-scale innovative computational methods and modeling in all physical sciences will be featured. CiCP is a refereed, interdisciplinary journal edited by a group of world-wide leading scientists in diverse fields to guarantee its highest standard. The journal is published in a timely fashion in printed and electronic forms.
It is important to understand how dropout, a popular regularization method, aids in achieving a good generalization solution during neural network training. In this work, we present a theoretical derivation of an implicit regularization of dropout, which is validated by a series of experiments. Additionally, we numerically study two implications of the implicit regularization, which intuitively rationalizes why dropout helps generalization. First, we find that input weights of hidden neurons tend to condense on isolated orientations trained with dropout. Condensation is a feature in the non-linear learning process, which makes the network less complex. Second, we find that the training with dropout leads to the neural network with a flatter minimum compared with standard gradient descent training, and the implicit regularization is the key to finding flat solutions. Although our theory mainly focuses on dropout used in the last hidden layer, our experiments apply to general dropout in training neural networks. This work points out a distinct characteristic of dropout compared with stochastic gradient descent and serves as an important basis for fully understanding dropout.
We study the reconstruction of structural connectivity for a general class of pulse-coupled nonlinear networks and show that the reconstruction can be successfully achieved through linear Granger causality (GC) analysis. Using spike-triggered correlation of whitened signals, we obtain a quadratic relationship between GC and the network couplings, thus establishing a direct link between the causal connectivity and the structural connectivity within these networks. Our work may provide insight into the applicability of GC in the study of the function of general nonlinear networks.