8 April 2025 in IEEE Journal on Emerging and Selected Topics in Circuits and Systems
As large language models (LLMs) continue to demonstrate exceptional capabilities across various domains, the challenge of achieving energy-efficient and accurate inference becomes increasingly critical. This work presents LightRot, a lightweight rotation scheme and dedicated hardware accelerator designed for low-bit LLM inference. The proposed architecture integrates Grouped Local Rotation (GLR) and Outlier Direction Aligning (ODA) algorithms with a hierarchical Fast Hadamard Transform (FHT)-based rotation unit to address key challenges in low-bit quantization, including the energy overhead of rotation operations. The proposed accelerator, implemented in a 28nm CMOS process, achieves a peak energy efficiency of 27.4 TOPS/W for 4-bit inference, surpassing prior state-of-the-art designs. Unlike conventional approaches that rely on higher-precision inference or evaluate on basic language modeling tasks like GPT-2, LightRot is optimized for advanced models such as LLaMA2-13B and LLaMA3-8B. Its performance is further validated on MT-Bench, demonstrating robust applicability to real-world conversational scenarios and redefining benchmarks for chat-based AI systems. By synergizing algorithmic innovations and hardware efficiency, this work sets a new paradigm for scalable, low-bit LLM inference, paving the way for sustainable AI advancements.
22 September 2025 in IEEE Transactions on Very Large Scale Integration (VLSI) Systems
Token merging is a widely adopted technique in vision transformers (ViTs) to reduce computation with minimal accuracy degradation. However, in tasks that require fine-grained outputs, such as text-unified object recognition, aggressive merging reaches a limit due to accuracy loss. To address this, we introduce a method that exploits residual token similarity beyond the merging threshold using difference computing, which significantly improves energy efficiency. The proposed processor integrates the post merging residual difference (PMRD) algorithm, which first reduces 34% of tokens through merging, then applies difference computing to the remaining tokens. This results in a 31% reduction in effective bit operations by leveraging residual similarity. To further exploit the resulting bit-level sparsity, the processor employs sparsity-separated accumulation with sign-magnitude representation (SSA-SM) and a bi-directional dynamic memory allocation (BDA) to optimize both computation and memory access. These enhancements yield an additional 29.4% computational energy savings. Overall, the system achieves a $2.22\times $ improvement in energy efficiency and delivers 7.23 TOPS/W on the MS COCO dataset for text-unified object recognition.
1 January 2026 in IEEE Journal of Solid-State Circuits