15 January 2024 in IEEE Transactions on Circuits and Systems I: Regular Papers
Large language models (LLMs) have sparked a new revolution in the field of natural language processing (NLP), and have garnered tremendous attention in both academic research and everyday life, thanks to their unprecedented performance in a wide range of applications. However, their deployment remains a significant challenge, primarily due to their intensive computational and memory requirements. Hardware acceleration and efficient quantization are promising solutions to address the two issues. In this paper, a quantization and hardware architecture co-design is presented for matrix-vector multiplications (MVMs) of LLMs. During quantization, we uniformly group weights and activations to ensure workload balance for hardware. To enhance the performance of quantization, we further propose two approaches called channel sorting and channel selection, which can be applied simultaneously. To support the proposed quantization scheme, we develop two precision-scalable MVM hardware architectures. They are specifically designed for high speed and high energy efficiency, respectively. Experimental results show that our proposed quantization scheme achieves state-of-the-art performance among all the reported post-training schemes that quantize both weights and activations into integers. Compared to MVM architecture of the state-of-the-art LLM accelerator OliVe, our design exhibits significant advantages in terms of area efficiency and energy efficiency.
22 December 2022 in IEEE Transactions on Circuits and Systems II: Express Briefs
Precision-scalable deep neural network (DNN) accelerator designs have attracted much research interest. Since the computation of most DNNs is dominated by multiply-accumulate (MAC) operations, designing efficient precision-scalable MAC (PSMAC) units is of central importance. This brief proposes two low-complexity PSMAC unit architectures based on the well-known one, Fusion Unit (FU), which is composed of a few basic units called Bit Bricks (BBs). We first simplify the architecture of BB through optimizing some redundant logic. Then a top-level architecture for PSMAC unit is devised by recursively employing BBs. Accordingly, two low-complexity PSMAC unit architectures are presented for two different kinds of quantization schemes. Moreover, we provide an insight into the decomposed multiplications and further reduce the bitwidths of the two architectures. Experimental results show that our proposed architectures can save up to 44.18% area cost and 45.45% power consumption when compared with the state-of-the-art design.
31 August 2023 in IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems
To meet the demand in a wide range of practical applications, precision-scalable deep neural network (DNN) accelerators are becoming an unavoidable trend. On the other hand, it has been demonstrated that a DNN accelerator may achieve better computation efficiency through exploiting the sparsity. Therefore, DNN accelerators with both precision scalability and sparsity exploitation are expected to have better performance. In this article, we propose an efficient precision-scalable DNN accelerator that can exploit the sparsity of activations. The precision scalability is obtained from the decomposable multiplier which is inspired by the well-known design, Bit Fusion. Besides, a zero-skipping scheme is adopted to leverage the inherent sparsity of activations. We first modify the architecture of the conventional fusion unit (FU) to make it amenable to the zero-skipping scheme. Then, a segmentation approach is devised to tackle the memory access conflict. Furthermore, a sparsity-aware mapping method is proposed to balance the workload of processing elements (PEs). Moreover, we present a bit-splitting strategy which can take advantage of the sparsity in the bit level. Compared with the state-of-the-art precision-scalable designs, our proposed accelerator can provide speedups of $4.12\times $ , $4.07\times $ , and $6.62\times $ in the precision modes $8b\times 8b$ , $4b\times 4b$ , and $2b\times 2b$ , respectively. Meanwhile, it also achieves $3.92\times $ peak area efficiency and competitive peak energy efficiency.
22 February 2024 in IEEE Transactions on Nanotechnology
Stochastic computing (SC) has emerged as a promising technique for reducing hardware costs in various applications, particularly in multiply-accumulate (MAC) intensive tasks such as neural networks. However, conventional SC still faces challenges in terms of achieving high accuracy and throughput. To enhance the precision, Sobol bit-stream has been widely adopted in SC. On the other hand, the throughput is frequently increased by means of parallel computing architecture. Nevertheless, directly increasing parallelism will incur significant additional hardware costs. In this paper, we propose Pseudo-Sobol bit-streams based on which an efficient parallel stochastic computing architecture for MAC operations is further developed. The proposed design leverages the properties of Pseudo-Sobol bit-streams and integrates the computation and conversion units to improve hardware efficiency. We evaluate the effectiveness of our design in two typical applications, general matrix multiplication (GEMM) and convolution. Experimental results show that our proposed design is capable of increasing energy efficiency by up to 36% and area efficiency by up to 70%.
9 May 2024 in IEEE Transactions on Computers
As a primary component of Transformers, attention mechanism suffers from quadratic computational complexity. To achieve efficient implementations, its hardware accelerator designs have aroused great research interest. However, most existing accelerators only support a single type of application and a single type of attention, making it difficult to meet the demands of diverse application scenarios. Additionally, they mainly focus on the dynamic pruning of attention matrices, which requires the deployment of pre-processing units, thereby reducing overall hardware efficiency. This paper presents CoDA which is an algorithm, dataflow and architecture co-design framework for versatile and efficient attention accelerators. The designed accelerator supports both NLP and CV applications, and can be configured into the mode supporting low-rank attention or low-rank plus sparse attention. We apply algorithmic transformations to low-rank attention to significantly reduce computational complexity. To prevent an increase in storage overhead resulting from the proposed algorithmic transformations, we carefully design the dataflows and adopt a block-wise fashion. Down-scaling softmax is further supported by architecture and dataflow co-design. Moreover, we propose a softmax sharing strategy to reduce the area cost. Our experiment results demonstrate that the proposed accelerator outperforms the state-of-the-art designs in terms of throughput, area efficiency and energy efficiency.