Xiaoyi Lu

Xiaoyi Lu

Auto-generated profile, based on publicly available information

Xiaoyi Lu

University of Florida, Gainesville, USA

Research Focus & Metrics

Academic Career Length

17Years (2010-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
1 February 2023 in Journal of Computer Science and Technology
Verified
Journal Article
Closed access
PDF file
AWAdam Weingram
Adam Weingram
YLYuke Li
Yuke Li
HQHao Qi
Hao Qi
DNDarren Ng
Darren Ng
LDLiuyao Dai
Liuyao Dai
XLXiaoyi Lu
Xiaoyi Lu

30 May 2024
Verified
Conference Paper
Open access
JHJiajun Huang
Jiajun Huang
SDSheng Di
Sheng Di
XYXiaodong Yu
Xiaodong Yu
YZYujia Zhai
Yujia Zhai
JLJinyang Liu
Jinyang Liu
RTRajeev Thakur
Rajeev Thakur
GPU-aware collective communication has become a major bottleneck for modern computing platforms as GPU computing power rapidly rises. A traditional approach is to directly integrate lossy compression into GPU-aware collectives, which can lead to serious performance issues such as underutilized GPU devices and uncontrolled data distortion. In order to address these issues, in this paper, we propose gZCCL, a first-ever general framework that designs and optimizes GPU-aware, compression-enabled collectives with an accuracy-aware design to control error propagation. To validate our framework, we evaluate the performance on up to 512 NVIDIA A100 GPUs with real-world applications and datasets. Experimental results demonstrate that our gZCCL-accelerated collectives, including both collective computation (Allreduce) and collective data movement (Scatter), can outperform NCCL as well as Cray MPI by up to 4.5 × and 28.7 ×, respectively. Furthermore, our accuracy evaluation with an image-stacking application confirms the high reconstructed data quality of our accuracy-aware framework.

26 August 2018 in IEEE Transactions on Parallel and Distributed Systems
Verified
Journal Article
Closed access
CCChing-Hsiang Chu
Ching-Hsiang Chu
XLXiaoyi Lu
Xiaoyi Lu
AAAmmar A. Awan
Ammar A. Awan
HSHari Subramoni
Hari Subramoni
BEBracy Elton
Bracy Elton
DPDhabaleswar K. Panda
Dhabaleswar K. Panda
Broadcast is a widely used operation in many streaming and deep learning applications to disseminate large amounts of data on emerging heterogeneous High-Performance Computing (HPC) systems. However, traditional broadcast schemes do not fully utilize hardware features for Graphics Processing Unit (GPU)-based applications. In this paper, a model-oriented analysis is presented to identify performance bottlenecks of existing broadcast schemes on GPU clusters. Next, streaming-based broadcast schemes are proposed to exploit InfiniBand hardware multicast (IB-MCAST) and NVIDIA GPUDirect technology for efficient message transmission. The proposed designs are evaluated in the context of using Message Passing Interface (MPI) based benchmarks and applications. The experimental results indicate improved scalability and up to 82 percent reduction of latency compared to the state-of-the-art solutions in the benchmark-level evaluation. Furthermore, compared to the state-of-the-art, the proposed design yields stable higher throughput for a synthetic streaming workload, and 1.3x faster training time for a deep learning framework.

11 June 2018 in IEEE Transactions on Multi-Scale Computing Systems
Verified
Journal Article
Closed access
XLXiaoyi Lu
Xiaoyi Lu
HSHaiyang Shi
Haiyang Shi
RBRajarshi Biswas
Rajarshi Biswas
MJM. Haseeb Javed
M. Haseeb Javed
DPDhabaleswar K. Panda
Dhabaleswar K. Panda
D eep L earning o ver B ig D ata (DLoBD) is an emerging paradigm to mine value from the massive amount of gathered data. Many Deep Learning frameworks, like Caffe, TensorFlow, etc., start running over Big Data stacks, such as Apache Hadoop and Spark. Even though a lot of activities are happening in the field, there is a lack of comprehensive studies on analyzing the impact of RDMA-capable networks and CPUs/GPUs on DLoBD stacks. To fill this gap, we propose a systematical characterization methodology and conduct extensive performance evaluations on four representative DLoBD stacks (i.e., CaffeOnSpark, TensorFlowOnSpark, MMLSpark/CNTKOnSpark, and BigDL) to expose the interesting trends regarding performance, scalability, accuracy, and resource utilization. Our observations show that RDMA-based design for DLoBD stacks can achieve up to 2.7x speedup compared to the IPoIB-based scheme. The RDMA scheme also scales better and utilizes resources more efficiently than IPoIB. For most cases, GPU-based schemes can outperform CPU-based designs, but we see that for LeNet on MNIST, CPU + MKL can achieve better performance than GPU and GPU + cuDNN on 16 nodes. Through our evaluation and an in-depth analysis on TensorFlowOnSpark, we find that there are large rooms to improve the designs of current-generation DLoBD stacks.

30 January 2024 in IEEE Micro
Verified
Journal Article
Closed access
LDLiuyao Dai
Liuyao Dai
HQHao Qi
Hao Qi
WCWeicong Chen
Weicong Chen
XLXiaoyi Lu
Xiaoyi Lu
Large Language Models (LLMs) like GPT, BERT, and T5 are pivotal in natural language processing. Their distributed training is influenced by high-speed interconnects. This study characterizes their training performance across various interconnects and communication protocols: TCP/IP, IPoIB, and RDMA, using data and model parallelism. RDMA-100Gbps outperforms IPoIB-100Gbps and TCP/IP-10Gbps, with average gains of 2.5x and 4.8x in data parallelism, while in model parallelism, the gains were 1.1x and 1.2x. RDMA achieves the highest interconnect utilization (up to 60 Gbps), compared to IPoIB with up to 20 Gbps and TCP/IP with up to 9 Gbps. Larger models demand increased communication bandwidth, with AllReduce in data parallelism consuming up to 91% of training time, and forward receive and Back-Embedding AllReduce in model parallelism taking up to 90%. The larger-scale experiment confirms that communication predominates iterations. Our findings underscore the significance of communication in distributed LLM training and present opportunities for optimization.

1 September 2013
Verified
Conference Paper
Closed access
RSRong Shi
Rong Shi
SPSreeram Potluri
Sreeram Potluri
KHKhaled Hamidouche
Khaled Hamidouche
XLXiaoyi Lu
Xiaoyi Lu
KTKaren Tomko
Karen Tomko
DPDhabaleswar K. Panda
Dhabaleswar K. Panda
Accelerating High-Performance Linkpack (HPL) on heterogeneous clusters with multi-core CPUs and GPUs has attracted a lot of attention from the High Performance Computing community. It is becoming common for large scale clusters to have GPUs on only a subset of nodes in order to limit system costs. The major challenge for HPL in this case is to efficiently take advantage of all the CPU and GPU resources available on a cluster. In this paper, we present a novel two-level workload partitioning approach for HPL that distributes workload based on the compute power of CPU/GPU nodes across the cluster. Our approach also handles multi-GPU configurations. Unlike earlier approaches for heterogeneous clusters with CPU and GPU nodes, our design takes advantage of asynchronous kernel launches and CUDA copies to overlap computation and CPU-GPU data movement. It uses techniques such as process grid reordering to reduce MPI communication/contention while ensuring load balance across nodes. Our experimental results using 32 GPU and 128 CPU nodes of Oakley, a research cluster at Ohio Supercomputer Center, shows that our proposed approach can achieve more than 80% of combined actual peak performance of CPU and GPU nodes. This provides 47% and 63% increase in the HPL performance that can be reported using only CPU nodes and only GPU nodes, respectively.