We describe the design motivation, architecture, deployment, and early operations of Expanse, a 5 Petaflop, heterogenous HPC system that entered production as an NSF-funded resource in December 2020 and will be operated on behalf of the national community for five years. Expanse will serve a broad range of computational science and engineering through a combination of standard batch-oriented services, and by extending the system to the broader CI ecosystem through science gateways, public cloud integration, support for high throughput computing, and composable systems. Expanse was procured, deployed, and put into production entirely during the COVID-19 pandemic, adhering to stringent public health guidelines throughout. Nevertheless, the planned production date of October 1, 2020 slipped by only two months, thanks to thorough planning, a dedicated team of technical and administrative experts, collaborative vendor partnerships, and a commitment to getting an important national computing resource to the community at a time of great need.
Voyager is an innovative computational resource designed by the San Diego Supercomputer Center in collaboration with technology partners to accelerate the development and performance of artificial intelligence and machine learning applications in science and engineering. Based on Intel’s Habana Labs first-generation deep learning (Gaudi) training and (Goya) inference processors, Voyager is funded by the National Science Foundation’s Advanced Computing Systems & Services Program as a Category II system and will be operated for 5 years, starting with an initial 3-year exploratory test-bed phase that will be followed by a 2-year allocated production phase for the national research community. Its AI-focused hardware features several innovative components, including fully-programmable tensor processing cores, high-bandwidth memory, and integrated, on-chip RDMA over Converged Ethernet network interfaces. In addition, Habana’s SynapseAI software suite provides seamless integration to popular machine learning frameworks like PyTorch and TensorFlow for end users. Here, we describe the design motivation for Voyager, its system architecture, software and user environment, initial benchmarking results, and the early science use cases and applications currently being ported to and deployed on the system.
Unique scientific instruments designed and operated by large global collaborations are expected to produce Exabyte-scale data volumes per year by 2030. These collaborations depend on globally distributed storage and compute to turn raw data into science. While all of these infrastructures have batch scheduling capabilities to share compute, Research and Education networks lack those capabilities. There is thus uncontrolled competition for bandwidth between and within collaborations. As a result, data "hogs" disk space at processing facilities for much longer than it takes to process, leading to vastly over-provisioned storage infrastructures. Integrated co-scheduling of networks as part of high-level managed workflows might reduce these storage needs by more than an order of magnitude. This paper describes such a solution, demonstrates its functionality in the context of the Large Hadron Collider (LHC) at CERN, and presents the next-steps towards its use in production.