25 August 2022 in IEEE Journal of Solid-State Circuits
Computing-in-memory (CIM) is an attractive approach for energy-efficient deep neural network (DNN) processing, especially for low-power edge devices. However, today’s typical DNNs usually exceed CIM-static random access memory (SRAM) capacity. The introduced off-chip communication covers up the benefits of CIM technique, meaning that CIM processors still encounter the memory bottleneck. To eliminate this bottleneck, we propose a CIM processor, called TT@CIM, which applies the tensor-train decomposition (TTD) method to compress the entire DNN to fit within CIM-SRAM. However, the cost of storage reduction by TTD is to introduce multiple serial small-size matrix multiplications, resulting in massive inefficient multiply-and-accumulate (MAC) and quantization operations (QuantOps). To achieve high energy efficiency, three optimization techniques are proposed in TT@CIM. First, TTD-CIM-matched dataflow is proposed to maximize CIM utilization and minimize additional MAC operations. Second, a bit-level-sparsity-optimized CIM macro with high bit-level-sparsity encoding scheme is designed to reduce the power consumption of one MAC operation. Third, a variable precision quantization method and a lookup table-based quantization unit are presented to improve the performance and energy efficiency of QuantOp. Fabricated in 28-nm CMOS and tested on 4/8-bit decomposed DNNs, TT@CIM achieves 5.99-to-691.13-TOPS/W peak energy efficiency depending on the operating voltage.
15 March 2023 in IEEE Transactions on Circuits and Systems I: Regular Papers
Skeleton-based human action cognition (HAR) has drawn increasing attention recently. As an emerging approach for skeleton-based HAR tasks, Spatial-Temporal Graph Convolution Network (STGCN) achieves remarkable performance by fully exploiting the skeleton topology information via graph convolution. Unfortunately, existing GCN accelerators lose efficiency when processing STGCN models due to two limitations (1) At the dataflow level, the hardware parallelism of GCN accelerators cannot match the computation parallelism of STGCN models, leading to computing resource under-utilization. (2) At the computation level, GCN accelerators fail to exploit the inherent temporal redundancy in STGCN models. To overcome the limitations, this paper proposes STAR, an STGCN architecture for skeleton-based human action recognition. STAR is designed based on the characteristics of different computation phases in STGCN. For limitation (1), a spatial-temporal dimension consistent (STDC) dataflow is proposed to fully exploit the data reuse opportunities in all the different dimensions of STGCN. For limitation (2), we propose a node-wise exponent sharing scheme and a temporal-structured redundancy elimination mechanism, to exploit the inherent temporal redundancy specially introduced by STGCN. To further address the under-utilization induced by redundancy elimination, we design a dynamic data scheduler to manage the feature data storage and schedule the features and weights for valid computation in real time. STAR achieves 4.48 $\times$ , 5.98 $\times$ , 2.54 $\times$ , and 103.88 $\times$ energy savings on average over the HyGCN, AWB-GCN, TPU, and Jetson TX2 GPU.
11 July 2022 in IEEE Transactions on Circuits and Systems I: Regular Papers
Deep neural network (DNN) has recently attracted tremendous attention in various fields. But the computing operation requirement and the memory bottleneck limit the energy efficiency of hardware implementations. Binary quantization is proposed to relieve the pressure of hardware design. And the Computing-In-Memory (CIM) is regarded as a promising method to resolve the memory wall challenge. However, the binary computing paradigm is mismatched with the CIM scheme, which incurs complex circuits and peripheral to realize binary operation in previous works. To overcome previous issues, this work presents Binary Representation Computation-In-Memory (BR-CIM) with several key features. (1) A lightweight computation unit is realized within the 6T SRAM array to accelerate binary computing and enlarge signal margin; (2) The reconfigurable computing scheme and mapping method support extendable bit precision to satisfy the accuracy requirement of various applications; (3) Simultaneous computing and weight loading is supported by column circuitry, which shortens the data loading latency; Several experiments are conducted to estimate algorithm accuracy, the computing latency, and power consumption. The energy efficiency reaches up to 1280 TOPs/W for binary representation. And the algorithm accuracy achieves 97.82%/76.4% on MNIST/CIFAR-100 dataset.
2 August 2024 in IEEE Journal of Solid-State Circuits
Transformer models have achieved impressive performance in various applications by effectively capturing contextual knowledge from the entire sequence. However, the multi-headed self-attention (MHSA) mechanism of Transformer models introduces multiple rounds of matrix multiplication (MM) and Softmax operations, which results in massive data movement and computations. Compute-in-memory (CIM) is a promising candidate to reduce data movement in the memory hierarchy of artificial intelligence (AI) accelerators, increasing the speed and energy efficiency for MM computation. However, the attention mechanism introduces dynamic MMs involving Query ( $Q$ ), Key ( $K$ ), and Value ( $V$ ). Since these matrices are both generated dynamically in previous layers, the dynamic MM mismatches the CIM paradigm, resulting in significant energy/latency consumption. This article proposes a CIM-based transformer accelerator (TranCIM) with three design features, effectively handling dynamic MMs. First, a correlative CIM ring (CRCIMR) executes the dynamic MM that involving Q and K by matrix decomposition, removing the loading of dynamically generated matrix in SRAM-based CIM (SRAM-CIM) cells. Second, a Softmax-based speculation unit (SSU) reduces the computation redundancy in dynamic MMs. Third, a digital-assisted CIM array (DACIMA) executes the dynamic MM that involving V based on symmetrical-L-shaped products, allowing the CIM macro to work in compute mode continuously. Fabricated in a 28-nm CMOS technology, the proposed accelerator occupies an area of 7.08 mm $^2$ . Measured on TinyBERT and BERT-Base with INT8 precision, the proposed accelerator achieves a system-level energy efficiency of 28.8 TOPS/W.
6 August 2025 in IEEE Transactions on Circuits and Systems II: Express Briefs
Attention-based large language models (LLMs) have revolutionized the natural language processing (NLP). Despite their impressive effectiveness, the quadratic complexity of self-attention incurs heavy computational and memory burdens. Dynamic sparse attention techniques emerge as a solution, however, the introduced extra prediction stage, coupled with costly data memory access, diminishes their hardware efficiency. To address these limitations, this brief proposes BETA, a fine-grained algorithm-architecture co-design tailored for sparse attention. First, a bit-grained multi-round filter (BMF) prediction is proposed to unveil and eliminate redundant memory access hidden in the sparsity prediction stage. Second, an adaptive and lightweight max-based threshold selection (MTS) strategy is developed to work in concert with the bit-wise prediction process. Third, a bit-wise out-of-order execution (BOOE) scheme is employed to enhance hardware utilization during bit-wise prediction. Finally, an elaborate architecture is designed to translate the theoretical complexity reduction into practical performance improvement. Implementation results demonstrate that BETA achieves 5.4×, 6.5×, 1.8× improvements in energy efficiency than the state-of-the-art Transformer accelerators Sanger, Spatten and SOFA, respectively, while maintaining comparable inference accuracy.