Transformer models achieve excellent results in the fields like natural language processing, computer vision, and bioinformatics. Their large numbers of matrix multiplications (MMs) lead to substantial data movement and computation. Although computing-in-memory (CIM) has proven to be an efficient architecture for MM computation, transformer’s attention mechanism raises new challenges in memory access and computation aspects: the dynamic MM in attention layers causes redundant OFF-chip memory access; Attention layers dominate transformer’s computation and require high precision. Thus, we design a bitline-transpose CIM-based transformer accelerator TranCIM with pipeline/parallel reconfigurable modes. The pipeline mode alleviates off-chip access for attention layers. The parallel mode is used by fully-connected (FC) layers for high parallelism. The full-digital CIM supports INT16 for attention layers and INT8 for FC layers, without analog CIM’s nonideal issues. Moreover, a sparse attention scheduler (SAS) is proposed to reduce attention computation. The fabricated TranCIM chip only consumes 15.59 $\mu \text{J}$ /Token for the bidirectional encoder representations from transformer (BERT)-base model, achieving $12.08\times $ – $36.82\times $ lower energy than prior CIM-based accelerators.

Cloud AI acceleration has drawn great attention in recent years, as big models are becoming a popular trend in deep learning. Cloud AI runs high-efficiency inference, high-accuracy inference and training, in demand of flexible floating-point (FP)/integer (INT) multiply–accumulation (MAC) support Many computing-in-memory (CIM) processors have been proposed for efficient AI acceleration. They usually rely on analog CIM techniques that are only suitable for high-efficiency neural network (NN) inference with low-precision INT MAC support. Since cloud AI demands high efficiency, high accuracy, and high flexibility simultaneously, we propose an innovative architecture reconfigurable digital CIM (ReDCIM) that meets all three requirements. We design the first CIM-based cloud AI processor, ReDCIM, which constructs a unified FP/INT pipeline architecture based on exponent pre-alignment and reconfigurable in-memory accumulation. Bitwise in-memory Booth multiplication is proposed to reduce computation on CIM. The fabricated ReDCIM chip achieves a state-of-the-art energy efficiency of 29.2 TFLOPS/W at BF16 and 36.5 TOPS/W at INT8.

Multimodal Transformers are emerging artificial intelligence (AI) models that comprehend a mixture of signals from different modalities like vision, natural language, and speech. The attention mechanism and massive matrix multiplications (MMs) cause high latency and energy. Prior work has shown that a digital computing-in-memory (CIM) network can be an efficient architecture to process Transformers while maintaining high accuracy. To further improve energy efficiency, attention-token-bit hybrid sparsity in multimodal Transformers can be exploited. The hybrid sparsity significantly reduces computation, but the irregularity also harms CIM utilization. To fully utilize the attention-token-bit hybrid sparsity of multimodal Transformers, we design a digital CIM-based accelerator called MulTCIM with three corresponding features: The long reuse elimination dynamically reshapes the attention pattern to improve CIM utilization. The runtime token pruner (RTP) removes insignificant tokens, and the modal-adaptive CIM network (MACN) exploits symmetric modal overlapping to reduce CIM idleness. The effective bitwidth-balanced CIM (EBB-CIM) macro balances input bits across in-memory multiply-accumulations (MACs) to reduce computation time. The fabricated MulTCIM consumes only $2.24~\mu \text{J}$ /Token for the ViLBERT-base model, achieving $2.50\times - 5.91\times $ lower energy than previous Transformer accelerators and digital CIM accelerators.

Digital computing-in-memory (CIM) has demonstrated outstanding energy and area efficiency. Although digital CIM avoids errors along the in-memory computing path, it is indispensable to correct hard and soft errors in digital CIM’s SRAM cells like conventional SRAM for commercial products. Hard errors come from defects in chip fabrication. Soft errors result from bit flips in an SRAM cell during CIM computation, and cause multiply-accumulation (MAC) errors. However, traditional SRAM’s error correction scheme cannot be directly applied to digital CIM, which causes hardware overhead in hard error repair, and incompatibility and latency problems in soft error correction. To tackle the above challenges, we propose an error-tolerant digital CIM processor ETCIM with three corresponding features: 1) a fault-adaptive CIM initializer (FACI) and input feeder reduce the hard error repair overhead by up to $8.47{\times }$ power and $18.16{\times }$ area savings; 2) a blockwise error correction code (BW-ECC) CIM macro suits the digital CIM structure that integrates multiple weight blocks, and corrects soft MAC errors; and 3) a progressive cell error corrector (PCEC) hides soft cell error correction latency during computation, reducing the total latency by up to 96.8%. The fabricated ETCIM achieves a state-of-the-art energy efficiency with an error correction of 19.5 tera operations per second (TOPS)/W at INT8.