The Hopfield model and the Boltzmann machine are among the most popular examples of neural networks. The latter, widely used for classification and feature detection, is able to efficiently learn a generative model from observed data and constitutes the benchmark for statistical learning. The former, designed to mimic the retrieval phase of an artificial associative memory lays in between two paradigmatic statistical mechanics models, namely the Curie-Weiss and the Sherrington-Kirkpatrick, which are recovered as the limiting cases of, respectively, one and many stored memories. Interestingly, the Boltzmann machine and the Hopfield network, if considered to be two cognitive processes (learning and information retrieval), are nothing more than two sides of the same coin. In fact, it is possible to exactly map the one into the other. We will inspect such an equivalence retracing the most representative steps of the research in this field.
We consider a three-layer Sejnowski machine and show that features learnt via contrastive divergence have a dual representation as patterns in a dense associative memory of order . The latter is known to be able to Hebbian store an amount of patterns scaling as , where denotes the number of constituting binary neurons interacting wisely. We also prove that, by keeping the dense associative network far from the saturation regime (namely, allowing for a number of patterns scaling only linearly with , while ) such a system is able to perform pattern recognition far below the standard signal-to-noise threshold. In particular, a network with is able to retrieve information whose intensity is even in the presence of a noise in the large limit. This striking skill stems from a redundancy representation of patterns—which is afforded given the (relatively) low-load information storage—and it contributes to explain the impressive abilities in pattern recognition exhibited by new-generation neural networks. The whole theory is developed rigorously, at the replica symmetric level of approximation, and corroborated by signal-to-noise analysis and Monte Carlo simulations.
11 September 2020 in Journal of Physics A: Mathematical and Theoretical
In this paper we adapt the broken replica interpolation technique (developed by Francesco Guerra to deal with the Sherrington-Kirkpatrick model, namely a pairwise mean-field spin-glass whose couplings are i.i.d. standard Gaussian variables) in order to work also with the Hopfield model (i.e. a pairwise mean-field neural-network whose couplings are drawn according to Hebb’s learning rule): this is accomplished by grafting Guerra’s telescopic averages on the transport equation technique, recently developed by some of the authors. As an overture, we apply the technique to solve the Sherrington-Kirkpatrick model with i.i.d. Gaussian couplings centered at J0 and with finite variance J ; the mean J0 provides a ferromagnetic contribution to be detected in a noisy environment tuned by J , hence making this model a natural test-case to be investigated before addressing the Hopfield model. For both the models, an explicit expression of their quenched free energy in terms of their natural order parameters is obtained at the K th step ( K arbitrary, but finite) of replica-symmetry-breaking. In particular, for the Hopfield model, by assuming that the overlaps respect Parisi’s decomposition (following the ziqqurat ansatz ) and that the Mattis magnetization is self-averaging, we recover previous results obtained via replica-trick by Amit, Crisanti and Gutfreund (1RSB) and by Steffan and Kühn (2RSB).
In this work we approach attractor neural networks from a machine learning perspective: we look for optimal network parameters by applying a gradient descent over a regularized loss function. Within this framework, the optimal neuron-interaction matrices turn out to be a class of matrices which correspond to Hebbian kernels revised by a reiterated unlearning protocol. Remarkably, the extent of such unlearning is proved to be related to the regularization hyperparameter of the loss function and to the training time. Thus, we can design strategies to avoid overfitting that are formulated in terms of regularization and early-stopping tuning. The generalization capabilities of these attractor networks are also investigated: analytical results are obtained for random synthetic datasets, next, the emerging picture is corroborated by numerical experiments that highlight the existence of several regimes (i.e., overfitting, failure and success) as the dataset parameters are varied.