23 June 2017 in IEEE Transactions on Very Large Scale Integration (VLSI) Systems
The spatial locality and the temporal locality of workloads are the root causes for cache designs to overcome the memory wall problem. However, the real memory access behavior for each of these applications can be very different. It gives the opportunities to explore further performance improvement due to different cache organization requirements. To address this issue, a spatial and temporal locality-aware adaptive cache is proposed, which dynamically partitions the private last level cache bank as prefetch region or victim region at runtime to explore the locality characteristics. The prefetch region speculates the data blocks in subsequent addresses to exploit the spatial locality, while the victim region collects the evicted data blocks from the upper memory hierarchy to exploit the temporal locality. Fast data prefetch with prioritized dynamic buffer management and adaptive burst-aware routing is realized in the proposed hybrid burst-support network-on-chip (HBNoC). By combining the adaptive cache partition with HBNoC, the off-chip misses and the on-chip network usage are greatly reduced. Experimental results demonstrate that the proposed adaptive cache design reduces up to 25% off-chip misses and improves 11.3% performance on average compared with the prior design, respectively.
3 November 2023 in IEEE Transactions on Very Large Scale Integration (VLSI) Systems
General purpose graphics processing units (GPGPUs) have emerged as a convincing and pivotal computing platform for deep learning applications. However, the fundamental tensor computations for neural networks on GPGPUs are still restricted by the von Neumann bottleneck. The memory bandwidth and energy consumption of moving a large amount of neural network data between the memory hierarchy and computational units of GPGPUs dominate the overall computational cost. To address these challenges, this article proposes TensorCache to reconstruct memory architecture with static random-access memory (SRAM)-based In-Cache Computing for efficient tensor computations in GPGPUs. It provides an innovative digital SRAM processing-in-memory (PIM) solution by transforming the cache array into large-scale PIM units, effectively mitigating the significant performance and energy consumption losses caused by data movement. To enable efficient hardware-software co-design for TensorCache, a decoupled architecture-based SRAM-PIM macro (SPM) is introduced at the hardware level, supporting in-memory bit-parallel comparison (IMBC) and near-memory radix-4 booth encoder (NRBE) for efficient mixed-precision floating-point (FP) tensor computations. At the software level, a programming model leveraging the GPGPU’s flexible programmability is proposed to bridge the gap between application demands and mismatched hardware/software interfaces. Experimental evaluations demonstrate that TensorCache achieves up to 38.59 $\times$ speedup and 16.26 $\times$ throughput enhancement compared to GPU CUDA Cores. Furthermore, it attains an acceleration of up to 1.78 $\times$ and 3.87 $\times$ throughput improvement compared to GPU Tensor Cores, while saving power consumption in tensor computations by over 90% with a mere 21% chip area overhead.
18 May 2023 in Microelectronics Journal
18 July 2025 in IEEE Transactions on Circuits and Systems II: Express Briefs
The attention mechanism has become increasingly popular due to its ability to capture complex dependencies, enabling models like transformers to achieve remarkable performance in large language models (LLMs), computer vision, and other domains. However, the mechanism faces challenges such as low arithmetic intensity, leading to frequent data movement, and long sequence lengths, which introduce a large amount of redundant information. To mitigate both data movement and computational overhead in attention mechanisms, we propose a hybrid CAM-SRAM processing-in-memory architecture. By leveraging the parallel search and sort capabilities of content-addressable memory (CAM) arrays, we achieve dynamic fine-grained sparsification on features with varying variance, reducing the number of multiply-accumulate (MAC) operations in the matrix multiplication (MatMul). Furthermore, an approximate booth encoding is employed in our MAC unit to reduce the number of partial products and maintain the consistency of their signs. This eliminates the need for negation operations, simplifying the logic design. Experimental results show that, in different configurations, our feature-level sparsification scheme achieves over 80% sparsity with an acceptable accuracy drop. With sparsity up to 80%, our design achieves a performance of 0.252-1.26 TOPS and a power efficiency of 4.71-21.72 TOPS/W, operating at 1000 MHz on the TSMC 40nm process.