We study empirical scaling laws for language model performance on the cross-entropy loss. The loss scales as a power-law with model size, dataset size, and the amount of compute used for training, with some trends spanning more than seven orders of magnitude. Other architectural details such as network width or depth have minimal effects within a wide range. Simple equations govern the dependence of overfitting on model/dataset size and the dependence of training speed on model size. These relationships allow us to determine the optimal allocation of a fixed compute budget. Larger models are significantly more sample-efficient, such that optimally compute-efficient training involves training very large models on a relatively modest amount of data and stopping significantly before convergence.
Recent work has demonstrated that black hole thermodynamics and information loss/restoration in AdS3/CFT2 can be derived almost entirely from the behavior of the Virasoro conformal blocks at large central charge, with relatively little dependence on the precise details of the CFT spectrum or OPE coefficients. Here, we elaborate on the non-perturbative behavior of Virasoro blocks by classifying all ‘saddles’ that can contribute for arbitrary values of external and internal operator dimensions in the semiclassical large central charge limit. The leading saddles, which determine the naive semiclassical behavior of the Virasoro blocks, all decay exponentially at late times, and at a rate that is independent of internal operator dimensions. Consequently, the semiclassical contribution of a finite number of high-energy states cannot resolve a well-known version of the information loss problem in AdS3. However, we identify two infinite classes of sub-leading saddles, and one of these classes does not decay at late times.
21 September 2017 in Journal of High Energy Physics
We chart the breakdown of semiclassical gravity by analyzing the Virasoro conformal blocks to high numerical precision, focusing on the heavy-light limit correspond-ing to a light probe propagating in a BTZ black hole background. In the Lorentzian regime, we find empirically that the initial exponential time-dependence of the blocks transitions to a universal \( t-\frac{3}{2} \) power-law decay. For the vacuum block the transition occurs at \( t\approx \frac{\uppi c}{6{h}_L} \), confirming analytic predictions. In the Euclidean regime, due to Stokes phenomena the naive semiclassical approximation fails completely in a finite region enclosing the ‘forbidden singularities’. We emphasize that limitations on the reconstruction of a local bulk should ultimately stem from distinctions between semiclassical and exact correlators.
One can obtain exact information about Virasoro conformal blocks by analytically continuing the correlators of degenerate operators. We argued in recent work that this technique can be used to explicitly resolve information loss problems in AdS3/CFT2. In this paper we use the technique to perform calculations in the small 1/c ∝ GN expansion: (1) we prove the all-orders resummation of logarithmic factors \( \propto \frac{1}{c} \log\;z \) in the Lorentzian regime, demonstrating that 1/c corrections directly shift Lyapunov exponents associated with chaos, as claimed in prior work, (2) we perform another all-orders resummation in the limit of large c with fixed cz, interpolating between the early onset of chaos and late time behavior, (3) we explicitly compute the Virasoro vacuum block to order 1/c2 and 1/c3 with external dimensions fixed, corresponding to 2 and 3 loop calculations in AdS3, and (4) we derive the heavy-light vacuum blocks in theories with \( \mathcal{N} \) = 1, 2 superconformal symmetry.
25 November 2019 in Journal of High Energy Physics
Causality constrains the gravitational interactions of massive higher spin particles in both AdS and flat spacetime. We explore the extent to which these constraints apply to composite particles, explaining why they do not rule out macroscopic objects or hydrogen atoms. However, we find that they do apply to glueballs and mesons in confining large N gauge theories. Assuming such theories contain massive bound states of general spin, we find parametric bounds in (3 + 1) spacetime dimensions of the form $$ N\lesssim \frac{M_{Pl}}{\Lambda_{\mathrm{QCD}}} $$ relating N, the QCD scale, and the Planck scale. We also argue that a stronger bound replacing ∧QCD with the UV cut-off scale may be derived from eikonal scattering in flat spacetime.
17 February 2021 in Journal of High Energy Physics
We combine old and new quantum field theoretic arguments to show that any theory of stable or metastable higher spin particles can be coupled to gravity only when the gravity sector has a stringy structure. Metastable higher spin particles, free or interacting, cannot couple to gravity while preserving causality unless there exist higher spin states in the gravitational sector much below the Planck scale Mpl. We obtain an upper bound on the mass Λgr of the lightest higher spin particle in the gravity sector in terms of quantities in the non-gravitational sector. We invoke the CKSZ uniqueness theorem to argue that any weakly coupled UV completion of such a theory must have a gravity sector containing infinite towers of asymptotically parallel, equispaced, and linear Regge trajectories. Consequently, gravitational four-point scattering amplitudes must coincide with the closed string four-point amplitude for s, t ≫ 1, identifying Λgr as the string scale. Our bound also implies that all metastable higher spin particles in 4d with masses m ≪ Λgr must satisfy a weak gravity condition.