Yi Ding

Yi Ding

Auto-generated profile, based on publicly available information

Yi Ding

School of Information Science and Technology, Xiamen University Tan Kah Kee College, Zhangzhou, Fujian, China

Research Focus & Metrics

Academic Career Length

3Years (2024-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications

5 July 2024
Verified
Conference Paper
Open access
XXXinyi Xie
Xinyi Xie
YDYi Ding
Yi Ding
CJChao Jiang
Chao Jiang
THTianyi Hang
Tianyi Hang
TLTianrun Lu
Tianrun Lu
HyperAttention is a new attention mechanism proposed to address the inevitability of quadratic time complexity and the conditionality of linear time when dealing with extended contexts, which allows the algorithm to maintain the computational complexity in linear time despite having large entries or a large stability rank in the matrix. HyperAttention improves the speed of inference and training of the large language model, with only a slight performance degradation and a significant speedup compared to FlashAttention. In order to further improve the accuracy of HyperAttention while maintaining its speed, we optimised each module for the original algorithm to improve its efficiency. Our main contribution is to change the locally sensitive hash algorithm used in HyperAttention for detecting large values and instead use k-means for indexing, further reducing the model's perplexity. In addition, we add applications on chatglm3-6b and baichuan-7b as well as chatglm3-6b-32k which supports long contexts in practice to make up for the lack of informativeness of HyperAttention in terms of practical applications, and test results are given. The experiments reveal that although the improved algorithm has a longer and relatively stable time complexity, it significantly reduces perplexity. Notably, with 10 to 15 replacement layers, the increase in time complexity is less than the reduction in perplexity, achieving a favourable balance between performance and efficiency. Moreover, we also analyse the feasibility of other sampling methods to improve accuracy and discuss the strategy for selecting the patch layers to apply the fast algorithm.