26 June 2025 in ACM Transactions on Architecture and Code Optimization
Driven by the increasing demands of machine learning, heterogeneous systems combining CPUs and GPUs have emerged as the dominant architecture for parallel computing in recent years. To optimize memory management and data transfer between CPUs and GPUs, Nvidia GPUs have introduced unified virtual memory ( UVM ) and pinned memory ( PM ) over the last decade. UVM can avoid explicit memory copies and potentially overlap GPU kernel computations with CPU-GPU data transfer. PM ensures that data with high locality remains in the main memory, preventing it from being paged out. In addition to these two techniques, asynchronous memory copy ( Async Memcpy ) was introduced recently in Nvidia GPUs to improve the CPU-GPU pipeline further. By utilizing Async Memcpy , the data transfer from GPU global memory to shared memory can be overlapped with GPU computations, adding an additional stage to the CPU-GPU data transfer pipeline. A thorough performance analysis of how Async Memcpy affects the current UVM and PM CPU-GPU data transfer scheme is desired. In this paper, we provide performance implications of the combined effect of UVM , PM , and Async Memcpy , exploring which applications benefit from which combination of these features. We implement all these features on a suite of 25 workloads, including microbenchmarks and realworld applications. We observe an average performance gain of \(24\% \) when utilizing UVM and a \(34\% \) gain when employing PM on realworld applications, compared to not applying any data transfer optimization techniques. The performance benefits of Async Memcpy vary across different workloads. For workloads featuring extensive shared memory usage and high compute density (e.g., kmeans and lud ), Async Memcpy delivers around a 20% performance improvement over using UVM or PM alone. In other workloads like knn , we note a \(20\% \) performance degradation when using Async Memcpy . Furthermore, we conduct an in-depth investigation of the GPU kernel using performance counters to uncover the root causes of performance differences among various data transfer models. We also perform sensitivity analyses to examine how the number of blocks and threads, as well as the L1-cache/shared memory partitioning, impact performance. We explore future research directions aimed at enhancing the data transfer pipeline by overlapping memory allocation with data transfer and computation across GPU kernels.
10 July 2025 in IEEE Computer Architecture Letters
Memory allocators, though constituting a small portion of the entire program code, can significantly impact application performance by affecting global factors such as cache behaviors. Moreover, memory allocators are often regarded as a “datacenter tax” inherent to all programs. Even a 1% improvement in performance can lead to significant cost and energy savings when scaled across an entire datacenter fleet. Modern memory allocators are designed to optimize allocation speed and memory fragmentation in multi-threaded environments, relying on complex metadata and control logic to achieve high performance. However, the overhead introduced by this complexity prompts a reevaluation of allocator design. Notably, such overhead can be avoided in single-threaded scenarios, which continue to be widely used across diverse application domains. In this paper, we present ExGen-Malloc, a memory allocator specifically optimized for single-threaded applications. We prototyped ExGen-Malloc, on a real system and demonstrated that it achieves a geometric mean speedup of 1.19 × over dlmalloc and 1.03 × over mimalloc, a modern multi-threaded allocator developed by Microsoft, on the SPEC CPU2017 benchmark suite.