We study a natural Wasserstein gradient flow on manifolds of probability distributions with discrete sample spaces. We derive the Riemannian structure for the probability simplex from the dynamical formulation of the Wasserstein distance on a weighted graph. We pull back the geometric structure to the parameter space of any given probability model, which allows us to define a natural gradient flow there. In contrast to the natural Fisher–Rao gradient, the natural Wasserstein gradient incorporates a ground metric on sample space. We illustrate the analysis of elementary exponential family examples and demonstrate an application of the Wasserstein natural gradient to maximum likelihood estimation.

We improve recently published results about resources of restricted Boltzmann machines (RBM) and deep belief networks (DBN) required to make them universal approximators. We show that any distribution [ ] on the set [ ] of binary vectors of length [ ] can be arbitrarily well approximated by an RBM with [ ] hidden units, where [ ] is the minimal number of pairs of binary vectors differing in only one entry such that their union contains the support set of [ ]. In important cases this number is half the cardinality of the support set of [ ] (given in Le Roux & Bengio, 2008 ). We construct a DBN with [ ], hidden layers of width [ ] that is capable of approximating any distribution on [ ] arbitrarily well. This confirms a conjecture presented in Le Roux and Bengio ( 2010 ).

We generalize recent theoretical work on the minimal number of layers of narrow deep belief networks that can approximate any probability distribution on the states of their visible units arbitrarily well. We relax the setting of binary units (Sutskever & Hinton, 2008 ; Le Roux & Bengio, 2008 , 2010 ; Montúfar & Ay, 2011 ) to units with arbitrary finite state spaces and the vanishing approximation error to an arbitrary approximation error tolerance. For example, we show that a q-ary deep belief network with L > or = 2 + (q[m-delta]-1 / (q-1)) layers of width n < or = + log(q) (m) + 1 for some [Formula : see text] can approximate any probability distribution on {0, 1, ... , q-1}n without exceeding a Kullback-Leibler divergence of delta. Our analysis covers discrete restricted Boltzmann machines and naive Bayes models as special cases.

We present a framework for designing cheap control architectures of embodied agents. Our derivation is guided by the classical problem of universal approximation, whereby we explore the possibility of exploiting the agent’s embodiment for a new and more efficient universal approximation of behaviors generated by sensorimotor control. This embodied universal approximation is compared with the classical non-embodied universal approximation. To exemplify our approach, we present a detailed quantitative case study for policy models defined in terms of conditional restricted Boltzmann machines. In contrast to non-embodied universal approximation, which requires an exponential number of parameters, in the embodied setting we are able to generate all possible behaviors with a drastically smaller model, thus obtaining cheap universal approximation. We test and corroborate the theory experimentally with a six-legged walking machine. The experiments indicate that the controller complexity predicted by our theory is close to the minimal sufficient value, which means that the theory has direct practical implications. Given a body and an environment, what is the brain complexity needed in order to generate a desired set of behaviors? The general understanding is that the physical properties of the body and the environment correlate with the required brain complexity. More precisely, it has been pointed that naturally evolved intelligent systems tend to exploit their embodiment constraints and that this allows them to express complex behaviors with relatively concise brains. Although this principle of parsimonious control has been formulated quite some time ago, only recently one has begun to develop the formalism that is required for making quantitative statements on the sufficient brain complexity given embodiment constraints. In this work we propose a precise mathematical approach that links the physical and behavioral constraints of an agent to the required controller complexity. As controller architecture we choose a well-known artificial neural network, the conditional restricted Boltzmann machine, and define its complexity as the number of hidden units. We conduct experiments with a virtual six-legged walking creature, which provide evidence for the accuracy of the theoretical predictions.

We compare two statistical models of three binary random variables. One is a mixture model and the other is a product of mixtures model called a restricted Boltzmann machine. Although the two models we study look different from their parametrizations, we show that they represent the same set of distributions on the interior of the probability simplex, and are equal up to closure. We give a semi-algebraic description of the model in terms of six binomial inequalities and obtain closed form expressions for the maximum likelihood estimates. We briefly discuss extensions to larger models.