Xiuhong Li

Xiuhong Li

Auto-generated profile, based on publicly available information

Xiuhong Li

Peking University, Beijing, China

Research Focus & Metrics

Academic Career Length

10Years (2017-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications

14 August 2024 in IEEE Transactions on Parallel and Distributed Systems
Verified
Journal Article
Open access
JDJiangfei Duan
Jiangfei Duan
XLXiuhong Li
Xiuhong Li
PXPing Xu
Ping Xu
XZXingcheng Zhang
Xingcheng Zhang
SYShengen Yan
Shengen Yan
DLDahua Lin
Dahua Lin
DNN models are becoming increasingly larger to achieve unprecedented accuracy, and the accompanying increased computation and memory requirements necessitate the employment of massive clusters and elaborate parallelization strategies to accelerate DNN training. In order to better optimize the performance and analyze the cost, it is indispensable to model the training throughput of distributed DNN training. However, complex parallelization strategies and the resulting complex runtime behaviors make it challenging to construct an accurate performance model. In this article, we present Proteus, the first standalone simulator to model the performance of complex parallelization strategies through simulation execution. Proteus first models complex parallelization strategies with a unified representation named Strategy Tree . Then, it compiles the strategy tree into a distributed execution graph and simulates the complex runtime behaviors, comp-comm overlap and bandwidth sharing , with a H ierarchical T opo- A ware E xecutor ( HTAE ). We finally evaluate Proteus across a wide variety of DNNs on three hardware configurations. Experimental results show that Proteus achieves 3.0% average prediction error and preserves order for training throughput of various parallelization strategies. Compared to state-of-the-art approaches, Proteus reduces prediction error by up to 133.8%.

30 March 2025
Verified
Conference Paper
Open access
RGRuihao Gong
Ruihao Gong
SBShihao Bai
Shihao Bai
SWSiyu Wu
Siyu Wu
YFYunqian Fan
Yunqian Fan
ZWZaijun Wang
Zaijun Wang
Xianglong LiuXianglong Liu
Xianglong Liu

24 November 2017 in IEEE Transactions on Computers
Verified
Journal Article
Closed access
XXXiaolong Xie
Xiaolong Xie
YLYun Liang
Yun Liang
XLXiuhong Li
Xiuhong Li
YWYudong Wu
Yudong Wu
Guangyu SunGuangyu Sun
Guangyu Sun
DFDongrui Fan
Dongrui Fan
The key to the high performance on GPUs lies in the massive threading to enable thread switching and hide long latencies. GPUs are equipped with a large register file to enable fast context switch. How- ever, thread throttling techniques that are designed to mitigate cache contention, lead to under-utilization of registers. Register allocation is a significant factor for performance as it not just determines the single- thread performance, but indirectly affects the TLP. In this paper, we propose Coordinated Register Allocation and Thread-level parallelism (CRAT), to explore the optimization space of register allocation and TLP management on GPUs. CRAT employs both compile-time(CRAT-static) and run-time techniques(CRAT-dyn) to exhaust the design space. Experimental results indicate that CRAT-static achieves an average 1.25X speedup over existing TLP management technique. On four register-limited applications, CRAT-dyn further improves the performance speedup of CRAT-static from 1.51X to 1.70X.

21 February 2020 in IEEE Transactions on Computers
Verified
Journal Article
Open access
LJLiancheng Jia
Liancheng Jia
YLYun Liang
Yun Liang
XLXiuhong Li
Xiuhong Li
LLLiqiang Lu
Liqiang Lu
SYShengen Yan
Shengen Yan
Modern CNNs require huge amount of convolution operations. To address the overwhelming computation problem, Winograd and FFT fast algorithms have been used as effective approaches to reduce the number of multiplications. Inputs and filters are transformed to special domains then perform element-wise multiplication, which can be transformed into batched GEMM operation. Different stages of computation contain multiple tasks with different computation and memory behaviors, and they share intermediate data. This provides the opportunity to fuse these tasks into a monolithic kernel. In this paper, we propose a kernel fusion technique for fast convolution algorithms based on MegaKernels. Each GPU thread block is assigned with one computation stage and we design a parameterized task mapping algorithm to assign stages to thread blocks. We build a task scheduler which fetches and executes the tasks following the task dependency. Evaluation on modern CNNs shows that our techniques achieve an average of 1.25X and 1.7X speedup compared to cuDNN's two implementations on Winograd convolution algorithm.

26 April 2026
Verified
Conference Paper
Open access
KHKe Hong
Ke Hong
XLXiuhong Li
Xiuhong Li
MLMinxu Liu
Minxu Liu
QMQiuli Mao
Qiuli Mao
TWTianqi Wu
Tianqi Wu
YWYu Wang
Yu Wang
Generative models have achieved remarkable success across various applications, driving the demand for multi-GPU computing. Inter-GPU communication becomes a bottleneck in multi-GPU computing systems, particularly on consumer-grade GPUs. By exploiting concurrent hardware execution, overlapping computation and communication latency becomes an effective technique for mitigating the communication overhead. We identify that an efficient and adaptable overlapping design should satisfy (1) tile-wise overlapping to maximize the overlapping opportunity, (2) interference-free computation to maintain the original computational performance, and (3) communication agnosticism to reduce the development burden against varying communication primitives. Nevertheless, current designs fail to simultaneously optimize for all of those features. To address the issue, we propose an overlapping design, named FlashOverlap, characterized by tile-wise overlapping, interference-free computation, and communication agnosticism. FlashOverlap utilizes a novel signaling mechanism: when part of the output finishes, the computation kernel sends a signal to trigger the communication of that part, while continuing the computation of the remaining part (interference-free computation). Consequently, the communication of the finished part and the computation of the remaining part can be overlapped. On top of the signaling mechanism, FlashOverlap comprises two key components: (1) the determination of the signaling timing to boost the overlap efficiency (tile-wise overlapping), and (2) a pre-communication reordering to create the contiguous address for finished data, enabling communication by simply calling NCCL [32] APIs (communication agnosticism), and a post-communication reordering to correct the data order. Experiments show that FlashOverlap achieves up to 1.65× speedup through overlap, outperforming existing works in most cases. Code is available at https://github.com/infinigence/FlashOverlap.