Tyler Griggs

Tyler Griggs

Auto-generated profile, based on publicly available information

Tyler Griggs

UC Berkeley

Research Focus & Metrics

Academic Career Length

2Years (2025-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
30 March 2025
Verified
Conference Paper
Open access
SCShiyi Cao
Shiyi Cao
SLShu Liu
Shu Liu
TGTyler Griggs
Tyler Griggs
PSPeter Schafhalter
Peter Schafhalter
XLXiaoxuan Liu
Xiaoxuan Liu
ISIon Stoica
Ion Stoica
Efficient deployment of large language models, particularly Mixture of Experts (MoE) models, on resource-constrained platforms presents significant challenges in terms of computational efficiency and memory utilization. The MoE architecture, renowned for its ability to increase model capacity without a proportional increase in inference cost, greatly reduces the token generation latency compared with dense models. However, the large model size makes MoE models inaccessible to individuals without high-end GPUs. In this paper, we propose a high-throughput MoE batch inference system, MoE-Lightning, that significantly outperforms past work. MoE-Lightning introduces a novel CPU-GPU-I/O pipelining schedule, CGOPipe, with paged weights to achieve high resource utilization, and a performance model, HRM, based on a Hierarchical Roofline Model we introduce to help find policies with higher throughput than existing systems. MoE-Lightning can achieve up to (10.3x) higher throughput than state-of-the-art offloading-enabled LLM inference systems for Mixtral 8x7B on a single T4 GPU (16GB). When the theoretical system throughput is bounded by the GPU memory, MoE-Lightning can reach the throughput upper bound with 2-3x less CPU memory, significantly increasing resource utilization. MoE-Lightning also supports efficient batch inference for much larger MoEs (e.g., Mixtral 8x22B and DBRX) on multiple low-cost GPUs (e.g., 2--4 T4s).

30 March 2025
Verified
Conference Paper
Open access
ZMZiming Mao
Ziming Mao
TXTian Xia
Tian Xia
ZWZhanghao Wu
Zhanghao Wu
WCWei-Lin Chiang
Wei-Lin Chiang
TGTyler Griggs
Tyler Griggs
ISIon Stoica
Ion Stoica