Ziheng Jiang

Ziheng Jiang

Auto-generated profile, based on publicly available information

Ziheng Jiang

ByteDance Seed, Seattle, USA

Research Focus & Metrics

Academic Career Length

2Years (2025-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
27 August 2025
Verified
Conference Paper
Open access
RZRuidong Zhu
Ruidong Zhu
ZJZiheng Jiang
Ziheng Jiang
CJChao Jin
Chao Jin
PWPeng Wu
Peng Wu
CSCesar A. Stuardo
Cesar A. Stuardo
XLXin Liu
Xin Liu

30 March 2025
Verified
Conference Paper
Open access
RLRuihang Lai
Ruihang Lai
JSJunru Shao
Junru Shao
SFSiyuan Feng
Siyuan Feng
SLSteven Lyubomirsky
Steven Lyubomirsky
BHBohan Hou
Bohan Hou
TCTianqi Chen
Tianqi Chen

26 April 2026
Verified
Conference Paper
Open access
CJChao Jin
Chao Jin
ZJZiheng Jiang
Ziheng Jiang
ZBZhihao Bai
Zhihao Bai
ZZZheng Zhong
Zheng Zhong
JLJuncai Liu
Juncai Liu
XLXin Liu
Xin Liu
We present MegaScale-MoE, a production system tailored for the efficient training of large-scale mixture-of-experts (MoE) models. MoE emerges as a promising architecture to scale large language models (LLMs) to unprecedented sizes, thereby enhancing model performance. However, existing MoE training systems experience a degradation in training efficiency, exacerbated by the escalating scale of MoE models and the continuous evolution of hardware. Recognizing the pivotal role of efficient communication in enhancing MoE training, MegaScale-MoE customizes communication-efficient parallelism strategies for attention and FFNs in each MoE layer and adopts a holistic approach to overlap communication with computation at both inter- and intra-operator levels. Additionally, MegaScale-MoE applies communication compression with adjusted communication patterns to lower precision, further improving training efficiency. When training a 352B MoE model on 1,440 NVIDIA Hopper GPUs, MegaScale-MoE achieves a training throughput of 1.41M tokens/s, improving the efficiency by 1.88X compared to Megatron-LM. We share our operational experience in accelerating MoE training and hope that by offering our insights in system design, this work will motivate future research in MoE systems.

22 March 2026
Verified
Conference Paper
Open access
ZZZiyi Zhang
Ziyi Zhang
ZJZiheng Jiang
Ziheng Jiang
CJChengquan Jiang
Chengquan Jiang
MYMenghan Yu
Menghan Yu
SZSize Zheng
Size Zheng
HHHenry Hoffmann
Henry Hoffmann