Zhichao Li

Zhichao Li

Auto-generated profile, based on publicly available information

Zhichao Li

Bytedance Inc

Research Focus & Metrics

Academic Career Length

4Years (2023-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
16 September 2024
Verified
Conference Paper
Open access
XWXinpeng Wei
Xinpeng Wei
ZLZhichao Li
Zhichao Li
CTCheng Tan
Cheng Tan
The trade-off of throughput versus latency is important in serving deep neural networks (DNNs) on GPUs. A hardware feature---Multi-Instance GPU (MIG)---provides an additional dimension to this trade-off. This paper studies the GPU sharing problem for serving DNNs with MIG. We propose a system Jormungandr that allocates MIG-enabled GPUs to optimize a user-define utility function, and places DNN serving containers on heterogeneous MIG instances. Jormungandr introduces a utility-first search that yields efficient solutions in practice. Additionally, it presents a robust transition protocol that guarantees a seamless switch between two GPU-cluster configurations, minimizing throughput fluctuations. We implement Jormungandr on Kubernetes. Our experiments show that Jormungandr provides near-optimal solutions, using < 6% more GPUs than the optimal solutions.

5 June 2023
Verified
Conference Paper
Open access
HLHeting Liu
Heting Liu
ZLZhichao Li
Zhichao Li
CTCheng Tan
Cheng Tan
RYRongqiu Yang
Rongqiu Yang
GCGuohong Cao
Guohong Cao
CGChuanxiong Guo
Chuanxiong Guo
Graphics processing units (GPUs) are the de facto standard for processing deep learning (DL) tasks. In large-scale GPU clusters, GPU failures are inevitable and may cause severe consequences. For example, GPU failures disrupt distributed training, crash inference services, and result in service level agreement violations. In this paper, we study the problem of predicting GPU failures using machine learning (ML) models to mitigate their damages. We train prediction models on a four-month production dataset with 350 million entries at ByteDance. We observe that classic prediction models (GBDT, MLP, LSTM, and 1D-CNN) do not perform well---they are inaccurate for predictions and unstable over time. We propose several techniques to improve the precision and stability of predictions, including parallel and cascade model-ensemble mechanisms and a sliding training method. We evaluate the performance of our proposed techniques. The results show that our proposed techniques improve the prediction precision from 46.3% to 85.4% on production workloads.