Rui Pan

Rui Pan

Auto-generated profile, based on publicly available information

Rui Pan

Princeton University, Princeton, USA

Research Focus & Metrics

Academic Career Length

3Years (2024-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
12 October 2025
Verified
Conference Paper
Open access
SRSiddhant Ray
Siddhant Ray
RPRui Pan
Rui Pan
ZGZhuohan Gu
Zhuohan Gu
KDKuntai Du
Kuntai Du
SFShaoting Feng
Shaoting Feng
JJJunchen Jiang
Junchen Jiang

4 November 2024
Verified
Conference Paper
Open access
YDYinwei Dai
Yinwei Dai
RPRui Pan
Rui Pan
AIAnand Iyer
Anand Iyer
KLKai Li
Kai Li
RNRavi Netravali
Ravi Netravali
Machine learning (ML) inference platforms are tasked with balancing two competing goals: ensuring high throughput given many requests, and delivering low-latency responses to support interactive applications. Unfortunately, existing platform knobs (e.g., batch sizes) fail to ease this fundamental tension, and instead only enable users to harshly trade off one property for the other. This paper explores an alternate strategy to taming throughput-latency tradeoffs by changing the granularity at which inference is performed. We present Apparate, a system that automatically applies and manages early exits (EEs) in ML models, whereby certain inputs can exit with results at intermediate layers. To cope with the time-varying overhead and accuracy challenges that EEs bring, Apparate repurposes exits to provide continual feedback that powers several novel runtime monitoring and adaptation strategies. Apparate lowers median response latencies by 40.5--91.5% and 10.0--24.2% for diverse CV and NLP classification workloads, and median time-per-token latencies by 22.6--77.9% for generative scenarios, without affecting throughputs or violating tight accuracy constraints.

18 December 2025 in IEEE Transactions on Networking
Verified
Journal Article
Closed access
JLJialong Li
Jialong Li
STShreyansh Tripathi
Shreyansh Tripathi
LRLakshay Rastogi
Lakshay Rastogi
YLYiming Lei
Yiming Lei
RPRui Pan
Rui Pan
YXYiting Xia
Yiting Xia
As machine learning models scale in size and complexity, their computational requirements become a significant barrier. Mixture-of-Experts (MoE) models alleviate this issue by selectively activating relevant experts. Despite this, MoE models are hindered by high communication overhead from all-to-all operations, low GPU utilization, and complications from heterogeneous GPU environments. This paper presents Comet, which optimizes both model deployment and all-to-all communication scheduling to address these challenges in MoE inference. Comet achieves minimal communication times by strategically ordering token transmissions in all-to-all communications. It improves GPU utilization by colocating experts from different models on the same device, avoiding the limitations of all-to-all communication. We analyze Comet’s optimization strategies theoretically across four common GPU cluster settings: exclusive vs. colocated models on GPUs, and homogeneous vs. heterogeneous GPUs. Comet provides optimal solutions for three cases, and for the remaining NP-hard scenario, it offers a polynomial-time sub-optimal solution with only a 1.09× degradation from the optimal, as shown in the simulation results. Comet is the first approach to minimize MoE inference time via optimal model deployment and communication scheduling across various scenarios. Evaluations demonstrate that Comet significantly accelerates inference, achieving speedups of up to 2.63× in homogeneous clusters and 2.91× in heterogeneous environments. Moreover, Comet enhances GPU utilization by up to 2.38× compared to existing methods.

4 November 2024
Verified
Conference Paper
Open access
AIAnand Padmanabha Iyer
Anand Padmanabha Iyer
MGMingyu Guan
Mingyu Guan
YDYinwei Dai
Yinwei Dai
RPRui Pan
Rui Pan
SGSwapnil Gandhi
Swapnil Gandhi
RNRavi Netravali
Ravi Netravali
Machine learning inference platforms continue to face high request rates and strict latency constraints. Existing solutions largely focus on compressing models to substantially lower compute costs (and time) with mild accuracy degradations. This paper explores an alternate (but complementary) technique that trades off accuracy and resource costs on a perinput granularity: early exit models, which selectively allow certain inputs to exit a model from an intermediate layer. Though intuitive, early exits face fundamental deployment challenges, largely owing to the effects that exiting inputs have on batch size (and resource utilization) throughout model execution. We present E3, the first system that makes early exit models practical for realistic inference deployments. Our key insight is to split and replicate blocks of layers in models in a manner that maintains a constant batch size throughout execution, all the while accounting for resource requirements and communication overheads. Evaluations with NLP and vision models show that E3 can deliver up to 1.74× improvement in goodput (for a fixed cost) or 1.78× reduction in cost (for a fixed goodput). Additionally, E3's goodput wins generalize to autoregressive LLMs (2.8--3.8×) and compressed models (1.67×).