1 July 2024 in Physical Review X
Deep learning algorithms demonstrate a surprising ability to learn high-dimensional tasks from limited examples. This is commonly attributed to the depth of neural networks, enabling them to build a hierarchy of abstract, low-dimensional data representations. However, how many training examples are required to learn such representations remains unknown. To quantitatively study this question, we introduce the random hierarchy model: a family of synthetic tasks inspired by the hierarchical structure of language and images. The model is a classification task where each class corresponds to a group of high-level features, chosen among several equivalent groups associated with the same class. In turn, each feature corresponds to a group of subfeatures chosen among several equivalent groups and so on, following a hierarchy of composition rules. We find that deep networks learn the task by developing internal representations invariant to exchanging equivalent groups. Moreover, the number of data required corresponds to the point where correlations between low-level features and classes become detectable. Overall, our results indicate how deep networks overcome the curse of dimensionality by building invariant representations and provide an estimate of the number of data required to learn a hierarchical task. Published by the American Physical Society 2024
27 January 2022 in Physical Review E
We put forward a general field theory for nearly flat fluid membranes with embedded activators and analyze their critical properties using renormalization group techniques. Depending on the membrane-activator coupling, we find a crossover between acoustic and diffusive scaling regimes, with mean-field dynamical critical exponents and 2, respectively. We argue that the acoustic scaling, which is exact in all spatial dimensions, leads to an early-time behavior, which is representative of the spatiotemporal patterns observed at the leading edge of motile cells, such as oscillations superposed on the growth of the membrane width. In the case of mean-field diffusive scaling, one-loop corrections to the mean-field exponents reveal universal behavior distinct from the Kardar-Parisi-Zhang scaling of passive interfaces and signs of strong-coupling behavior.
21 October 2024 in Journal of Statistical Mechanics: Theory and Experiment
Understanding how convolutional neural networks (CNNs) can efficiently learn high-dimensional functions remains a fundamental challenge. A popular belief is that these models harness the local and hierarchical structure of natural data such as images. Yet, we lack a quantitative understanding of how such structure affects performance, for example the rate of decay of the generalisation error with the number of training samples. In this paper, we study infinitely wide deep CNNs in the kernel regime. First, we show that the spectrum of the corresponding kernel inherits the hierarchical structure of the network, and we characterise its asymptotics. Then, we use this result together with generalisation bounds to prove that deep CNNs adapt to the spatial scale of the target function. In particular, we find that if the target function depends on low-dimensional subsets of adjacent input variables then the decay of the error is controlled by the effective dimensionality of these subsets. Conversely, if the target function depends on the full set of input variables then the error decay is controlled by the input dimension. We conclude by computing the generalisation error of a deep CNN trained on the output of another deep CNN with randomly initialised parameters. Interestingly, we find that, despite their hierarchical structure, the functions generated by infinitely wide deep CNNs are too rich to be efficiently learnable in high dimensions.
12 May 2020 in Physical Review Letters
Vital biological processes such as genome repair require fast and efficient binding of selected proteins to specific target sites on DNA. Here we propose an active target search mechanism based on “chromophoresis,” the dynamics of DNA-binding proteins up or down gradients in the density of epigenetic marks, or colors (biochemical tags on the genome). We focus on a set of proteins that deposit marks from which they are repelled—a case which is only encountered away from thermodynamic equilibrium. For suitable ranges of kinetic parameter values, chromophoretic proteins can perform undirectional motion and are optimally redistributed along the genome. Importantly, they can also locally unravel a region of the genome which is collapsed due to self-interactions and “dive” deep into its core, for a striking enhancement of the efficiency of target search on such an inaccessible substrate. We discuss the potential relevance of chromophoresis for DNA repair.
23 December 2025 in Physical Review E
How do neural language models acquire a language's structure when trained for next-token prediction? We address this question by deriving theoretical scaling laws for neural network performance on synthetic datasets generated by the random hierarchy model (RHM)-an ensemble of probabilistic context-free grammars designed to capture the hierarchical structure of natural language while remaining analytically tractable. Previously, we developed a theory of representation learning based on data correlations that explains how deep learning models capture the hierarchical structure of the data sequentially, one layer at a time. Here, we extend our theoretical framework to account for architectural differences. In particular, we predict and empirically validate that convolutional networks, whose structure aligns with that of the generative process through locality and weight sharing, enjoy a faster scaling of performance compared to transformer models, which rely on global self-attention mechanisms. This finding clarifies the architectural biases underlying neural scaling laws and highlights how representation learning is shaped by the interaction between model architecture and the statistical properties of data.
27 January 2022 in Physical Review E
Motivated by experimental observations of patterning at the leading edge of motile eukaryotic cells, we introduce a general model for the dynamics of nearly-flat fluid membranes driven from within by an ensemble of activators. We include, in particular, a kinematic coupling between activator density and membrane slope which generically arises whenever the membrane has a nonvanishing normal speed. We unveil the phase diagram of the model by means of a perturbative field-theoretical renormalization group analysis. Due to the aforementioned kinematic coupling the natural early-time dynamical scaling is acoustic, that is the dynamical critical exponent is 1. However, as soon as the the normal velocity of the membrane is tuned to zero, the system crosses over to diffusive dynamic scaling in mean field. Distinct critical points can be reached depending on how the limit of vanishing velocity is realized: in each of them corrections to scaling due to nonlinear coupling terms must be taken into account. The detailed analysis of these critical points reveals novel scaling regimes which can be accessed with perturbative methods, together with signs of strong coupling behavior, which establishes a promising ground for further nonperturbative calculations. Our results unify several previous studies on the dynamics of active membrane, while also identifying nontrivial scaling regimes which cannot be captured by passive theories of fluctuating interfaces and are relevant for the physics of living membranes.