17 August 2021 in IEEE Journal of Solid-State Circuits
Recent advances in deep neural networks (DNNs) and machine learning algorithms have induced the demand for services based on machine learning algorithms that require a large number of computations, and specialized hardware ranging from accelerators for data centers to on-device computing systems have been introduced. Low-precision math such as 8-bit integers have been used in neural networks for energy-efficient neural network inference, but training with low-precision numbers without performance degradation have remained to be a challenge. To overcome this challenge, this article presents an 8-bit floating-point neural network training processor for state-of-the-art non-sparse neural networks. As naïve 8-bit floating-point numbers are insufficient for training DNNs robustly, two additional methods are introduced to ensure high-performance DNN training. First, a novel numeric system which we dub as 8-bit floating point with shared exponent bias (FP8-SEB) is introduced. Moreover, multiple-way fused multiply-add (FMA) trees are used in FP8-SEB’s hardware implementation to ensure higher numerical precision and reduced energy. FP8-SEB format combined with multiple-way FMA trees is evaluated under various scenarios to show a trained-from-scratch performance that is close to or even surpasses that of current networks trained with full-precision (FP32). Our silicon-verified DNN training processor utilizes 24-way FMA trees implemented with FP8-SEB math and flexible 2-D routing schemes to show $2.48\times $ higher energy efficiency than prior low-power neural network training processors and $78.1\times $ lower energy than standard GPUs.
10 June 2025 in IEEE Transactions on Circuits and Systems I: Regular Papers
Transformer-based large language models (LLMs) excel in text generation but face challenges like memory bandwidth bottlenecks and large key-value (KV) cache sizes as context lengths grow, impacting low-latency performance. Existing accelerators adopt parallelism, model compression, and sparsity exploitation but often fail to fully utilize token-specific sparsity, limiting their effectiveness for long context lengths. CLAT addresses these issues with a low-overhead clustering algorithm that identifies relevant key vector clusters for each token’s query, omitting less relevant vectors with minimal impact. It optimizes memory bandwidth using routing for single-batch inference and introduces scheduling techniques to reduce attention layer latency. Additionally, CLAT compresses model parameters to 4-bit precision and KV caches to 8-bit precision, supported by a multi-precision MAC structure that avoids extra overhead. Validated on Llama2-7B, OPT-6.7B, and Llama3-8B models, CLAT reduces attention layer latency by up to 88.6% and overall text generation latency by up to 34.9%. It improves single-batch text generation throughput by $1.66\times $ to $2.42\times $ over an A100 GPU, demonstrating significant performance gains.
26 September 2024 in IEEE Journal of Solid-State Circuits
Speech enhancement (SE) plays a key role in many audio-related applications by removing noise and enhancing the quality of human voice. Recent deep learning-based approaches provide high-quality SE, but real-time processing of those algorithms is challenging in resource-constrained devices due to high computational complexity. In this article, we present an energy-efficient real-time SE processor aimed at hearing aids. To implement high-quality SE with a very limited power budget, various algorithm and hardware optimization techniques are proposed. Our SE algorithm adaptively allocates computational resources to each region in the input feature domain depending on their importance, reducing overall computations by 29.7%. Along with 4-bit channel-wise logarithmic quantization, the processor adopts a reconfigurable multiplier-less processing element (PE) that supports both pre-/post-processing and neural network layers, resulting in a 21.5% area reduction. In addition, the design employs efficient scheduling and input buffering schemes to reduce on-chip memory access by 70.8%. Fabricated in a 28-nm CMOS process, our design consumes only $740~{\mu }$ W at 2.5 MHz with a total latency of 39.96 ms, satisfying the real-time processing constraints. In addition, our approach demonstrated higher SE quality than prior art in both objective and subjective evaluations.
7 April 2026 in IEEE Transactions on Circuits and Systems I: Regular Papers
Recent point-based neural networks have shown promising performance in 3D object detection by directly processing raw point clouds without converting them into voxels. However, they exhibit stage-specific computational bottlenecks due to the varying computational demands of sampling, grouping, and neural network operations. This paper presents an area- and energy-efficient processor that accelerates point-based neural networks with low latency through algorithm-hardware co-optimization. The proposed feature-decoupled PointNet enables reusing point features by separating MLPs, reducing MAC operations by up to 67.6% without accuracy loss. In addition, density-aware adaptive block pruning eliminates redundant computations in overly dense regions, significantly lowering the computational costs of sampling and grouping. The proposed 3D object detection processor efficiently parallelizes the grouping process by combining a dual-core architecture with a lightweight prioritized shift register, effectively mitigating the workload imbalance. It also adopts a dual-center fused-layer dataflow that maximizes PE utilization while significantly reducing external memory access. Furthermore, the masked max pooling unit eliminates the need for a separate gathering unit by integrating the gathering step into the pooling logic for higher area and power efficiency. Experimental results show that the design provides real-time 3D object detection with a high throughput of 77.9 frames/s, also exhibiting a high accuracy of 83.11 mAP on the KITTI dataset, surpassing prior arts in both accuracy and throughput. In addition, its low energy consumption of 2.5 mJ/frame and high area efficiency of 24.0 frames/s/mm2 make it particularly well-suited for embedded systems for autonomous driving.
19 September 2025 in IEEE Transactions on Circuits and Systems II: Express Briefs
Voxel-based point cloud neural networks have gained significant attention for 3D semantic segmentation due to their effectiveness in processing point clouds. However, the high computational overhead of processing large-scale point clouds and the inherent irregularity of these point clouds hinder the fast and energy-efficient acceleration of segmentation. This brief presents a 3D semantic segmentation accelerator with an offset-wise weight quantization technique that drastically reduces computational complexity. The proposed design introduces a unified voxel search unit that efficiently processes various types of operations needed to capture spatial relationships among irregularly stored voxels. In addition, a dual-mode computing engine combined with a novel workload allocation technique enables highly parallel processing and maximizes processing core utilization. Fabricated in a 28-nm CMOS technology, the proposed processor achieves 10.24 TOPS/W energy efficiency, outperforming prior art.