Numerical relativity is central to the investigation of astrophysical sources in the dynamical and strong-field gravity regime, such as binary black hole and neutron star coalescences. Current challenges set by gravitational-wave and multimessenger astronomy call for highly performant and scalable codes on modern massively parallel architectures. We present GR-Athena++, a general-relativistic, high-order, vertex-centered solver that extends the oct-tree, adaptive mesh refinement capabilities of the astrophysical (radiation) magnetohydrodynamics code Athena++. To simulate dynamical spacetimes, GR-Athena++ uses the Z4c evolution scheme of numerical relativity coupled to the moving puncture gauge. We demonstrate stable and accurate binary black hole merger evolutions via extensive convergence testing, cross-code validation, and verification against state-of-the-art effective-one-body waveforms. GR-Athena++ leverages the task-based parallelism paradigm of Athena++ to achieve excellent scalability. We measure strong-scaling efficiencies above 95% for up to ∼1.2 × 104 CPUs and excellent weak scaling is shown up to ∼105 CPUs in a production binary black hole setup with adaptive mesh refinement. GR-Athena++ thus allows for the robust simulation of compact binary coalescences and offers a viable path toward numerical relativity at exascale.
1 June 2021 in Journal of Cosmology and Astroparticle Physics
We study the detailed process by which slow contraction smooths and flattens the universe using an improved numerical relativity code that accepts initial conditions with non-perturbative deviations from homogeneity and isotropy along two independent spatial directions. Contrary to common descriptions of the early universe, we find that the geometry first rapidly converges to an inhomogeneous, spatially-curved and anisotropic ultralocal state in which all spatial gradient contributions to the equations of motion decrease as an exponential in time to negligible values. This is followed by a second stage in which the geometry converges to a homogeneous, spatially flat and isotropic spacetime. In particular, the decay appears to follow the same history whether the entire spacetime or only parts of it are smoothed by the end of slow contraction.
12 February 2025 in The Astrophysical Journal Supplement Series
We present the extension of GR-Athena++ to general-relativistic magnetohydrodynamics (GRMHD) for applications to neutron star spacetimes. The new solver couples the constrained transport implementation of Athena++ to the Z4c formulation of the Einstein equations to simulate dynamical spacetimes with GRMHD using oct-tree adaptive mesh refinement. We consider benchmark problems for isolated and binary neutron star spacetimes demonstrating stable and convergent results at relatively low resolutions and without grid symmetries imposed. The code correctly captures magnetic field instabilities in nonrotating stars with total relative violation of the divergence-free constraint of 10−16. It handles evolutions with a microphysical equation of state and black hole formation in the gravitational collapse of a rapidly rotating star. For binaries, we demonstrate correctness of the evolution under the gravitational radiation reaction and show convergence of gravitational waveforms. We showcase the use of adaptive mesh refinement to resolve the Kelvin–Helmholtz instability at the collisional interface in a merger of magnetised binary neutron stars. GR-Athena++ shows strong scaling efficiencies above 80% in excess of 105 CPU cores and excellent weak scaling is shown up to ∼5 × 105 CPU cores in a realistic production setup. GR-Athena++ allows for the robust simulation of GRMHD flows in strong and dynamical gravity with exa-scale computers.
1 July 2024 in Journal of Cosmology and Astroparticle Physics
In a systematic study, we use an equivalent pair of improved numerical relativity codes based on a tetrad-formulation of the classical Einstein-scalar field equations to examine whether slow contraction or inflation (or both) can resolve the homogeneity, isotropy and flatness problems. Our finding, based on a set of gauge/frame invariant diagnostics and the models considered, is that slow contraction robustly and rapidly smooths and flattens spacetime beginning from initial conditions that are outside the perturbative regime of the flat Friedmann-Robertson-Walker metric, whereas inflation fails these tests. We present new numerical evidence supporting the conjecture that the combination of ultralocal evolution and an effective equation-of-state with pressure much greater than energy density is the key to having robust and rapid smoothing. The opposite of ultralocality occurs in expanding spacetimes, which is the leading obstruction to smoothing following a big bang.
14 January 2025 in The Astrophysical Journal Supplement Series
We introduce an extension to the AthenaK code for general-relativistic magnetohydrodynamics (GRMHD) in dynamical spacetimes using a 3+1 conservative Eulerian formulation. Like the fixed-spacetime GRMHD solver, we use standard finite-volume methods to evolve the fluid and a constrained-transport scheme to preserve the divergence-free constraint for the magnetic field. We also utilize a first-order flux correction (FOFC) scheme to reduce the need for an artificial atmosphere and optionally enforce a maximum principle to improve robustness. We demonstrate the accuracy of AthenaK using a set of standard tests in flat and curved spacetimes. Using a SANE accretion disk around a Kerr black hole, we compare the new solver to the existing solver for stationary spacetimes using the so-called “HARM-like” formulation. We find that both formulations converge to similar results. We also include the first published binary neutron star (BNS) mergers performed on graphical processing units (GPUs). Thanks to the FOFC scheme, our BNS mergers maintain a relative error of or better in baryon mass conservation up to collapse. Finally, we perform scaling tests of AthenaK on OLCF Frontier, where we show excellent weak scaling of ≥80% efficiency up to 32,768 GPUs and 74% up to 65,536 GPUs for a GRMHD problem in dynamical spacetimes with six levels of mesh refinement. AthenaK achieves an order-of-magnitude speedup using GPUs compared to CPUs, demonstrating that it is suitable for performing numerical relativity problems on modern exascale resources.
The numerical solution of relativistic hydrodynamics equations in conservative form requires root-finding algorithms that invert the conservative-to-primitive variables map. These algorithms employ the equation of state of the fluid and can be computationally demanding for applications involving sophisticated microphysics models, such as those required to calculate accurate gravitational wave signals in numerical relativity simulations of binary neutron stars. This work explores the use of machine learning methods to speed up the recovery of primitives in relativistic hydrodynamics. Artificial neural networks are trained to replace either the interpolations of a tabulated equation of state or directly the conservative-to-primitive map. The application of these neural networks to simple benchmark problems shows that both approaches improve over traditional root finders with tabular equation-of-state and multi-dimensional interpolations. In particular, the neural networks for the conservative-to-primitive map accelerate the variable recovery by more than an order of magnitude over standard methods while maintaining accuracy. Neural networks are thus an interesting option to improve the speed and robustness of relativistic hydrodynamics algorithms.