9 March 2026 in ACM Computing Surveys
The emergence of large-scale Mixture of Experts (MoE) models represents a significant advancement in artificial intelligence, offering larger model capacity and computational efficiency through conditional computation. However, deploying and running inference on these models presents significant challenges in computational resources, latency, and energy efficiency. This comprehensive survey analyzes optimization techniques for MoE models across the entire system stack. We first establish a taxonomical framework that categorizes optimization approaches into model-level, system-level, and hardware-level optimizations. At the model level, we examine architectural innovations including efficient expert design, attention mechanisms, various compression techniques such as pruning, quantization, and knowledge distillation, as well as algorithm improvement including dynamic routing strategies and expert merging methods. At the system level, we investigate distributed computing approaches, load balancing mechanisms, and efficient scheduling algorithms that enable scalable deployment. Furthermore, we delve into hardware-specific optimizations and co-design strategies that maximize throughput and energy efficiency. This survey provides both a structured overview of existing solutions and identifies key challenges and promising research directions in MoE inference optimization. To facilitate ongoing updates and the sharing of cutting-edge advances in MoE inference optimization research, we have established a repository accessible at https://github.com/MoE-Inf/awesome-moe-inference/ .
8 October 2026 in ACM Transactions on Architecture and Code Optimization
Retrieval-Augmented Generation (RAG) significantly improves Large Language Models (LLMs) but introduces massive input sequences that severely bottleneck the prefill stage. While KV-cache reuse reduces redundant computation for shared document prefixes, the reusable KV working set in RAG serving can exceed GPU memory capacity, requiring KV chunks to be retained across host DRAM and SSDs. However, naive multi-tier storage extensions suffer from severe I/O bottlenecks, suboptimal eviction, and high CPU-GPU data transfer overheads, which often negate the latency benefits of cache reuse. In this paper, we propose PCR, a P refetch-enhanced C ache R euse system for low-latency RAG serving. PCR transforms passive SSD-backed storage into an active, latency-hiding memory hierarchy through three core modules: (1) a Multi-Tier Prefix-Tree Cache Manager that unifies the organization and tracking of reusable KV chunks across the entire memory hierarchy of GPU, host DRAM, and SSD; (2) a Queue-Guided Runtime Scheduler that leverages the post-retrieval waiting queue as a look-ahead signal to proactively protect hot chunks and prefetch SSD-resident data into host DRAM before execution; and (3) a Latency-Hiding KV Transfer Pipeline that overlaps fine-grained PCIe data movement and asynchronous SSD operations with model computation. Extensive evaluations across diverse models and RAG workloads show that PCR reduces TTFT over the evaluated KV-cache reuse systems in most settings, with a 2.45 × speedup over vLLM for Llama3.1-8B on A6000 under Workload 1 at 1.0 request/s, while maintaining lower tail latency under high load.