Weihao Cui

Weihao Cui

Auto-generated profile, based on publicly available information

Weihao Cui

Shanghai Jiao Tong University, Shanghai, China

Research Focus & Metrics

Academic Career Length

8Years (2019-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
1 November 2019
Verified
Conference Paper
Closed access
WCWeihao Cui
Weihao Cui
MWMengze Wei
Mengze Wei
QCQuan Chen
Quan Chen
XTXiaoxin Tang
Xiaoxin Tang
JLJingwen Leng
Jingwen Leng
MGMingyi Guo
Mingyi Guo
GPUs have been widely adopted to serve online deep learning-based services that have stringent QoS requirements. However, emerging deep learning serving systems often result in long latency and low throughput of the inference requests that damage user experience and increase the number of GPUs required to host an online service. Our investigation shows that the poor batching operation and the lacking of data transfer-computation overlap are the root causes of the long latency and low throughput. To this end, we propose Ebird, a deep learning serving system that is comprised of a GPU-resident memory pool, a multi-granularity inference engine, and an elastic batch scheduler. The memory pool eliminates the unnecessary waiting of the batching operation and enables data transfer-computation overlap. The inference engine enables concurrent execution of different batches, improving the GPUs resource utilization. The batch scheduler organizes inference requests elastically. Our experimental results on an Nvidia Titan RTX GPU show that Ebird reduces the response latency of inferences by up to 70.9% and improves the throughput by up to 49.3% while guaranteeing the QoS target compared with TensorFlow Serving.

28 December 2020 in IEEE Transactions on Parallel and Distributed Systems
Verified
Journal Article
Closed access
WCWeihao Cui
Weihao Cui
QCQuan Chen
Quan Chen
HZHan Zhao
Han Zhao
MWMengze Wei
Mengze Wei
XTXiaoxin Tang
Xiaoxin Tang
MGMinyi Guo
Minyi Guo
We aim to tackle existing problems about deep learning serving on GPUs in the view of the system. GPUs have been widely adopted to serve online deep learning-based services that have stringent QoS(Quality-of-Service) requirements. However, emerging deep learning serving systems often result in poor responsiveness and low throughput of the inferences that damage user experience and increase the number of GPUs required to host an online service. Our investigation shows that the poor batching operation and the lack of data transfer-computation overlap are the root causes of the poor responsiveness and low throughput. To this end, we propose E $^2$ bird, a deep learning serving system that is comprised of a GPU-resident memory pool, a multi-granularity inference engine, and an elastic batch scheduler. The memory pool eliminates the unnecessary waiting of the batching operation and enables data transfer-computation overlap. The inference engine enables concurrent execution of different batches, improving the GPU resource utilization. The batch scheduler organizes inferences elastically to guarantee the QoS. Our experimental results on an Nvidia Titan RTX GPU show that E $^2$ bird reduces the response latency of inferences by up to 82.4 percent and improves the throughput by up to 62.8 percent while guaranteeing the QoS target compared with TensorFlow Serving.

8 March 2021 in IEEE Transactions on Computers
Verified
Journal Article
Closed access
WZWei Zhang
Wei Zhang
QCQuan Chen
Quan Chen
NZNingxin Zheng
Ningxin Zheng
WCWeihao Cui
Weihao Cui
KFKaihua Fu
Kaihua Fu
MGMinyi Guo
Minyi Guo
Datacenters use GPUs to provide the significant computing throughput required by emerging user-facing services. The diurnal user access pattern of user-facing services provides a strong incentive to co-located applications for better GPU utilization, and prior work has focused on enabling co-location on multicore processors and traditional non-preemptive GPUs. However, current GPUs are evolving towards spatial multitasking and introduce a new set of challenges to eliminate QoS violations. To address this open problem, we explore the underlying causes of QoS violation on spatial multitasking GPUs. In response to these causes, we propose C-Laius, a runtime system that carefully allocates the computation resource to co-located applications for maximizing the throughput of batch applications while guaranteeing the required QoS of user-facing services. C-Laius not only allows co-locating one user-facing application with multiple batch applications, but also supports the co-location of multiple user-facing applications with batch applications. In the case of a single co-located user-facing application, our evaluation on an Nvidia RTX 2080Ti GPU shows that C-Laius improves the utilization of spatial multitasking GPUs by 20.8 percent, while achieving the 99%-ile latency target for user-facing services. As to the case of multiple co-located user-facing applications, C-Laius ensures no violation of QoS while improving the accelerator utilization by 35.9 percent on average.

14 February 2024 in IEEE Transactions on Computers
Verified
Journal Article
Closed access
CGCong Guo
Cong Guo
FXFengchen Xue
Fengchen Xue
JLJingwen Leng
Jingwen Leng
YQYuxian Qiu
Yuxian Qiu
YGYue Guan
Yue Guan
MGMinyi Guo
Minyi Guo
Network pruning can reduce the computation cost of deep neural network (DNN) models. However, sparse models often produce randomly-distributed weights to maintain accuracy, leading to irregular computations. Consequently, unstructured sparse models cannot achieve meaningful speedup on commodity hardware built for dense matrix computations. Accelerators are usually modified or designed with structured sparsity-optimized architectures for exploiting sparsity. For example, the Ampere architecture introduces a sparse tensor core, which adopts the 2:4 sparsity pattern. We propose a pruning method that builds upon the insight that matrix multiplication generally breaks the large matrix into multiple smaller tiles for parallel execution. We present the “tile-wise” sparsity pattern, which maintains a structured sparsity pattern at the tile level for efficient execution but allows for irregular pruning at the global scale to maintain high accuracy. In addition, the tile-wise sparsity is implemented at the global memory level, and the 2:4 sparsity executes at the register level inside the sparse tensor core. We can combine these two patterns into a “tile-vector-wise” ( TVW ) sparsity pattern to explore more fine-grained sparsity and further accelerate the sparse DNN models. We evaluate the TVW on the GPU, achieving averages of 1.85 $\boldsymbol{\times}$ , 2.75 $\boldsymbol{\times}$ , and 22.18 $\boldsymbol{\times}$ speedups over the dense model, block sparsity, and unstructured sparsity.

30 March 2025
Verified
Conference Paper
Open access
SZShulai Zhang
Shulai Zhang
QCQuan Chen
Quan Chen
WCWeihao Cui
Weihao Cui
HZHan Zhao
Han Zhao
CXChunyu Xue
Chunyu Xue
MGMinyi Guo
Minyi Guo

12 October 2022 in IEEE Transactions on Computers
Verified
Journal Article
Open access
HZHan Zhao
Han Zhao
WCWeihao Cui
Weihao Cui
QCQuan Chen
Quan Chen
MGMinyi Guo
Minyi Guo
Emerging GPUs have multiple Streaming Multiprocessors (SM), while each SM is comprised of CUDA Cores and Tensor Cores. While CUDA Cores do the general computation, Tensor Cores are designed to speed up matrix multiplication for deep learning applications. However, a GPU kernel often either uses CUDA Cores or Tensor Cores, leaving the other processing units idle. Although many prior research works have been proposed to co-locate kernels to improve GPU utilization, they cannot leverage the Intra-SM CUDA Core-Tensor Core Parallelism. Specifically, ISPA designs persistent and elastic block to solve the thread slot and shared memory contention between co-located kernels. ISPA also adopts the register allocation method to manage the register contention. These resource management methods are applicable for both white-box kernels and $cudnn$ kernels. Experimental results on an Nvidia 2080Ti GPU show that ISPA improves the system-wide throughput by 15.3% for white-box workloads, and 7.1% for $cudnn$ -based workloads compared with prior co-location work.