Overlap of computation and communication is critical for good application-level performance. Modern high-performance networks offer Hardware-assisted tag matching and rendezvous offload to enable communication progress without involving the host CPU. However, hardware based offload cannot be used in many situations due to various hardware limitations and performance issues. Furthermore, hardware-based designs cannot provide good overlap for common communication patterns involving unexpected messages or non-contiguous datatypes. In this paper, we address these limitations by designing a communication-aware overlap engine for MPI that uses novel hardware-assisted and software-based solutions to extract overlap for both expected and unexpected messages. The proposed design adapts to the application’s communication requirements including message size, datatype, and relative timing of processes using heuristics and history-driven predictions. We evaluate the proposed designs against state-of-the-art MPI libraries and show up to 41% and 22% reduction in latency for collective operations and stencil-based application kernels on 1024 and 128 nodes, respectively, as well as 23% improvement in communication performance of the P3DFFT application.
The importance of GPUs in accelerating HPC applications is evident by the fact that a large number of supercomputing clusters are GPU-enabled. Many of these HPC applications use MPI as their programming model. These MPI applications frequently exchange data that is non-contiguous in GPU memory. MPI provides Derived Datatypes (DDTs) to represent such data. Past research on DDTs mainly focused on optimizing the pack-unpack kernels. Modern HCAs are capable of gathering/scattering data from/to non-contiguous GPU memory regions. We propose a low-overhead HCA-assisted scheme to improve the performance of GPU-based non-contiguous exchanges without the GPU-based pack-unpack kernels. We show that the proposed scheme provides up to 2X benefits compared to the existing pack-based scheme at the benchmark level. Furthermore, we show up to 17% improvement with the SW4Lite application compared to other MPI libraries such as MVAPICH2-GDR and OpenMPI+UCX.