Xiaolong Yang

Xiaolong Yang

Auto-generated profile, based on publicly available information

Xiaolong Yang

School of Integrated Circuits, Beijing Innovation Center for Future Chip, and Beijing National Research Center for Information Science and Technology, Tsinghua University, Beijing, China

Research Focus & Metrics

Academic Career Length

4Years (2023-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
17 June 2023
Verified
Conference Paper
Open access
YQYubin Qin
Yubin Qin
YWYang Wang
Yang Wang
DDDazheng Deng
Dazheng Deng
ZZZhiren Zhao
Zhiren Zhao
XYXiaolong Yang
Xiaolong Yang
SYShouyi Yin
Shouyi Yin
Transformer model is becoming prevalent in various AI applications with its outstanding performance. However, the high cost of computation and memory footprint make its inference inefficient. We discover that among the three main computation modules in a Transformer model (QKV generation, attention computation, FFN), it is the QKV generation and FFN that contribute to the most power cost. While the attention computation, focused by most previous works, only has decent power share when dealing with extremely long inputs. Therefore, in this paper, we propose FACT, an efficient algorithm-hardware co-design optimizing all three modules of Transformer. We first propose an eager prediction algorithm which predicts the attention matrix before QKV generation. It further detects the unnecessary computation in QKV generation and assigns mixed-precision FFN with the predicted attention, which helps improve the throughput. Further, we propose FACT accelerator to efficiently support eager prediction with three designs. It avoids the large overhead of prediction by using log-based add-only operations for prediction. It eliminates the latency of prediction through an out-of-order scheduler that makes the eager prediction and computation work in full pipeline. It additionally avoids memory access conflict in the mixed-precision FFN with a novel diagonal storage pattern. Experiments on 22 benchmarks show that our FACT improves the throughput of the whole Transformer by 3.59× on the geomean average. It achieves an enviable 47.64× and 278.1× energy saving when computing attention, compared to previous attention-optimization-only SOTA works ELSA and Sanger. Further, FACT achieves an energy efficiency of 4388 GOPS/W performing the whole Transformer layer on average, which is 94.98× higher than Nvidia V100 GPU.

23 June 2024
Verified
Conference Paper
Open access
XYXiaolong Yang
Xiaolong Yang
YWYang Wang
Yang Wang
YQYubin Qin
Yubin Qin
JWJiachen Wang
Jiachen Wang
SWShaojun Wei
Shaojun Wei
SYShouyi Yin
Shouyi Yin
With the continuous advancement of artificial intelligence, neural networks exhibit an escalating parameter size, demanding increased computational power and excessive memory access. Low bit-width quantization emerges as a viable solution to address this challenge. However, conventional low bit-width uniform quantization suffers from a mismatch with the weight and activation data distribution in neural networks, resulting in accuracy degradation. We propose Fibonacci Quantization, which matches the distribution of weights and activations by using Fibonacci numbers. It achieves negligible accuracy loss for ResNet50 on ImageNet1k with both activations and weights quantized to 4-bit. Based on the Fibonacci Quantization, we present the Fibonacci Quantization Processor. It comprises two types of multiplication-free computing units: the Dualistic-Transformation Adder (DTA) and the Bit-Exclusive Adder (BEA), both capable of transforming the multiplication of Fibonacci numbers into simple addition. In addition, to effectively map multiplications of small and large Fibonacci numbers onto BEA and DTA, we propose Topological-Order Routing (TOR) that routes data either to the previous or current position. Our 4-bit Fibonacci quantization achieves a 0.98% higher accuracy compared with 4-bit uniform quantization for ResNet50 on ImageNet1k. For equivalent accuracy, our proposed processor outperforms uniform quantization with 2.17× higher energy efficiency.

12 January 2026 in IEEE Transactions on Circuits and Systems I: Regular Papers
Verified
Journal Article
Closed access
XYXiaolong Yang
Xiaolong Yang
YWYang Wang
Yang Wang
YQYubin Qin
Yubin Qin
WXWende Xu
Wende Xu
RGRuiqi Guo
Ruiqi Guo
SYShouyi Yin
Shouyi Yin
Transformer models have achieved remarkable success in various domains. Given concerns about user privacy, there is an urgent need for on-device fine-tuning of Transformer models at the edge. Transformer fine-tuning faces three key challenges: 1) $O(n^{3})$ re-computations during BP/WG save only $O(n^{2})$ storage, limiting batch size for fine-tuning speedup. 2)Weakly related tokens account for 87.9% of computations but contribute only 8.7% to accuracy. 3)89.1% of multiplications in matrix multiplications (MM) involve dual near-zero operands, leading to a $1.9\times $ increase in logic toggling energy due to frequent exponent/mantissa variations near zero. This paper proposes a Transformer-based processor supporting energy-efficient fine-tuning with three key features to tackle the above challenges. 1)An exponent-stationary re-computing scheduler (ESRS) reduces 44.2% of the storage requirement for each batch. 2)An aggressive linear fitting unit (ALFU) saves 47.4% of the computations in each iteration. 3)A logarithmic domain processing element (LDPE) decreases 36.3% of energy for MM in fine-tuning. Fabricated with 22nm technology, the proposed processor has an area of 6.4 mm2. The proposed Transformer processor achieves a peak energy efficiency of 54.94 TFLOPS/W. It reduces fine-tuning energy by $4.27\times $ and offers $3.57\times $ speedup for GPT-2.