3 November 2026 in ACM Transactions on Design Automation of Electronic Systems
AI workloads increasingly demand high parallelism and efficient on-chip memory utilization on modern GPUs. However, the limited capacity of shared memory often constrains thread-level parallelism, while portions of the L2 cache remain underutilized. In this work, we propose SAI , a mechanism that virtualizes shared memory into the L2 cache to improve GPU performance for AI applications. SAI dynamically activates virtualization based on runtime resource usage, enabling additional CTAs with minimal architectural modifications. We further introduce an L2 cache management strategy that integrates associativity-based virtual page allocation and a replacement information table, reducing page-swapping overhead while preserving L2 cache performance. Experimental results demonstrate that SAI achieves an 18.6% performance improvement over the baseline design and outperforms SMILE-opt, our reproduced state-of-the-art baseline, by 8.8%. Moreover, SAI captures 60.2% of the performance gain delivered by the idealized Double SMEM design while reducing energy consumption to 87.7% of the baseline. These results highlight the effectiveness of SAI in enhancing thread-level parallelism and optimizing on-chip memory for AI workloads.
15 September 2025 in IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems
The victim cache was originally designed as a small-capacity secondary cache to hold recently evicted lines from the L1 data cache in CPUs. However, this design is often suboptimal for graphics processing units (GPUs) due to the following two specific features of GPUs. First, a significant number of GPU applications have low hit rates in the L1 data cache, while the probability of lines evicted from the L1 data cache being subsequently reused is high. This means that the number of memory requests that hit in the victim cache can approach or even exceed the number of requests that hit in the L1 data cache. Second, GPUs use a unified architecture for the L1 data cache and shared memory, or the L1 cache and texture cache, to save hardware overhead. However, this hardware architecture significantly increases the L1 data cache access latency. We observe that some applications are susceptible to access latency, and increasing latency can severely degrade performance. To this end, we introduce HIVE+, an enhanced high-priority victim cache designed to accelerate GPU memory accesses. The key idea of HIVE+ is to exploit idle registers as the data array of the victim cache and place the tag array of the victim cache at a higher level in the memory hierarchy than the L1 data cache. Our experimental results show that HIVE+ achieves an average performance improvement of 47.5% and 35.0% compared to the baseline (BL) and Linebacker architectures, respectively.
13 October 2026 in ACM Transactions on Architecture and Code Optimization
Sparse Matrix-Sparse Matrix Multiplication (SpMSpM) is a crucial computational kernel widely used in scientific computing and machine learning. The varying sparse patterns across different matrices pose significant challenges for conventional accelerators with fixed dataflow architectures. Although recent studies have explored dynamic dataflow approaches to better capture memory access patterns under diverse sparsity conditions, these solutions still struggle to simultaneously improve data reuse, load balance, and merging efficiency. To address these limitations, we present an SpMSpM accelerator based on adaptive dataflow and efficient merging (ADEM). ADEM is carefully designed from four key aspects. First, we propose the segmented fiber tree (SFT) data structure, which extends the conventional fiber tree through further partitioning to better support our dataflow paradigm while enhancing data reuse. We then present a cache-aware dataflow to mitigate memory overflow issues. Furthermore, ADEM decouples the multiplication and merging phases, utilizing the SFT structure to enable fine-grained task scheduling for improved load balancing. Finally, the accelerator incorporates heterogeneous merging units specifically optimized for handling two distinct types of merging operations, thereby significantly improving merger utilization. Compared with the state-of-the-art baseline system, ADEM achieves average speedups of 1.25×, 1.73×, and 1.16× on VGG-16, ResNet-50, and SuiteSparse workloads, respectively.