Yang Hu

Yang Hu

Auto-generated profile, based on publicly available information

Yang Hu

Tsinghua University

Research Focus & Metrics

Academic Career Length

8Years (2019-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
17 June 2023
Verified
Conference Paper
Open access
YQYubin Qin
Yubin Qin
YWYang Wang
Yang Wang
DDDazheng Deng
Dazheng Deng
ZZZhiren Zhao
Zhiren Zhao
XYXiaolong Yang
Xiaolong Yang
SYShouyi Yin
Shouyi Yin
Transformer model is becoming prevalent in various AI applications with its outstanding performance. However, the high cost of computation and memory footprint make its inference inefficient. We discover that among the three main computation modules in a Transformer model (QKV generation, attention computation, FFN), it is the QKV generation and FFN that contribute to the most power cost. While the attention computation, focused by most previous works, only has decent power share when dealing with extremely long inputs. Therefore, in this paper, we propose FACT, an efficient algorithm-hardware co-design optimizing all three modules of Transformer. We first propose an eager prediction algorithm which predicts the attention matrix before QKV generation. It further detects the unnecessary computation in QKV generation and assigns mixed-precision FFN with the predicted attention, which helps improve the throughput. Further, we propose FACT accelerator to efficiently support eager prediction with three designs. It avoids the large overhead of prediction by using log-based add-only operations for prediction. It eliminates the latency of prediction through an out-of-order scheduler that makes the eager prediction and computation work in full pipeline. It additionally avoids memory access conflict in the mixed-precision FFN with a novel diagonal storage pattern. Experiments on 22 benchmarks show that our FACT improves the throughput of the whole Transformer by 3.59× on the geomean average. It achieves an enviable 47.64× and 278.1× energy saving when computing attention, compared to previous attention-optimization-only SOTA works ELSA and Sanger. Further, FACT achieves an energy efficiency of 4388 GOPS/W performing the whole Transformer layer on average, which is 94.98× higher than Nvidia V100 GPU.

7 March 2024 in IEEE Circuits and Systems Magazine
Verified
Journal Article
Closed access
YHYang Hu
Yang Hu
XLXinhan Lin
Xinhan Lin
HWHuizheng Wang
Huizheng Wang
ZHZhen He
Zhen He
XYXingmao Yu
Xingmao Yu
SYShouyi Yin
Shouyi Yin
Nowadays, artificial intelligence (AI) technology with large models plays an increasingly important role in both academia and industry. It also brings a rapidly increasing demand for the computing power of the hardware. As the computing demand for AI continues to grow, the growth of hardware computing power has failed to keep up. This has become a significant factor restricting the development of AI. The augmentation of hardware computing power is mainly propelled by the escalation of transistor density and chip area. However, the former is impeded by the termination of the Moore’s Law and Dennard scaling, and the latter is significantly restricted by the challenge of disrupting the legacy fabrication equipment and process. In recent years, advanced packaging technologies that have gradually matured are increasingly used to implement bigger chips that integrate multiple chiplets, while still providing interconnections with chip-level density and bandwidth. This technique points out a new path of continuing the increase of computing power while leveraging the current fabrication process without significant disruption. Enabled by this technique, a chip can extend to a size of wafer-scale (over 10,000 mm $^{2}$ ), provisioning orders of magnitude more computing capabilities (several POPS within just one monolithic chip) and die-to-die bandwidth density (over 15 GB/s/mm) than regular chips, and emerges a new Wafer-scale Computing paradigm. Compared to conventional high-performance computing paradigms such as multi-accelerator and datacenter-scale computing, Wafer-scale Computing shows remarkable advantages in communication bandwidth, integration density, and programmability potential. Not surprisingly, disruptive Wafer-scale Computing also brings unprecedented design challenges for hardware architecture, design- $\backslash $ system- technology co-optimization, power and cooling systems, and compiler tool chain. At present, there are no comprehensive surveys summarizing the current state and design insights of Wafer-scale Computing. This article aims to take the first step to help academia and industry review existing wafer-scale chips and essential technologies in a one-stop manner. So that people can conveniently grasp the basic knowledge and key points, understand the achievements and shortcomings of existing research, and contribute to this promising research direction.

29 May 2024 in IEEE Journal of Solid-State Circuits
Verified
Journal Article
Closed access
RGRuiqi Guo
Ruiqi Guo
XCXiaofeng Chen
Xiaofeng Chen
LWLei Wang
Lei Wang
YWYang Wang
Yang Wang
HSHao Sun
Hao Sun
SYShouyi Yin
Shouyi Yin
Transformer models have achieved impressive performance in various artificial intelligence (AI) applications. However, the high cost of computation and memory footprint make its inference inefficient. Although digital compute-in-memory (CIM) is a promising hardware architecture with high accuracy, Transformer’s attention mechanism raises three challenges in the access and computation of CIM: 1) the attention computation involving Query and Key results in massive data movement and under-utilization in CIM macros; 2) the attention computation involving Possibility and Value exhibits plenty of dynamic bit-level sparsity, resulting in redundant bit-serial CIM operations; and 3) the restricted data reload bandwidth in CIM macros results in a significant decrease in performance for large Transformer models. To address these challenges, we design a CIM accelerator called CIM Transformer (CIMFormer) with three corresponding features. First, the token-pruning-aware attention reformulation (TPAR) is a technique that adjusts attention computations according to the token-pruning ratio. This reformulation reduces the real-time access to and under-utilization of CIM macros. Second, the principal possibility gather-scatter scheduler (PPGSS) gathers the possibilities with greater effective bit-width as concurrent inputs to CIM macros, enhancing the efficiency of bit-serial CIM operations. Third, the systolic X $\mid$ W-CIM macro array efficiently handles the execution of large Transformer models that exceed the storage capacity of the on-chip CIM macros. Fabricated in a 28-nm technology, CIMFormer achieves a peak energy efficiency of 15.71 TOPS/W, with an over 1.46 $\times$ improvement compared with the state-of-the-art Transformer accelerator at an equivalent situation.

20 June 2025
Verified
Conference Paper
Open access
ZXZheng Xu
Zheng Xu
DKDehao Kong
Dehao Kong
JLJiaxin Liu
Jiaxin Liu
JLJinxi Li
Jinxi Li
JHJingxiang Hou
Jingxiang Hou
SYShouyi Yin
Shouyi Yin

1 November 2020 in IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems
Verified
Journal Article
Closed access
Zhendong WangZhendong Wang
Zhendong Wang
ZJZihang Jiang
Zihang Jiang
ZWZhen Wang
Zhen Wang
XTXulong Tang
Xulong Tang
CLCong Liu
Cong Liu
YHYang Hu
Yang Hu

20 June 2025
Verified
Conference Paper
Open access
QYQize Yang
Qize Yang
TWTaiquan Wei
Taiquan Wei
SGSihan Guan
Sihan Guan
CLChengran Li
Chengran Li
HSHaoran Shang
Haoran Shang
YHYang Hu
Yang Hu