Zeyu Mi

Zeyu Mi

Auto-generated profile, based on publicly available information

Zeyu Mi

Institute of Parallel and Distributed Systems, Shanghai Jiao Tong University, Shanghai, China

Research Focus & Metrics

Academic Career Length

9Years (2018-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
4 November 2024
Verified
Conference Paper
Open access
YSYixin Song
Yixin Song
ZMZeyu Mi
Zeyu Mi
HXHaotong Xie
Haotong Xie
HCHaibo Chen
Haibo Chen
This paper introduces PowerInfer, a high-speed Large Language Model (LLM) inference engine on a personal computer (PC) equipped with a single consumer-grade GPU. The key principle underlying the design of PowerInfer is exploiting the high locality inherent in LLM inference, characterized by a power-law distribution in neuron activation. This distribution indicates that a small subset of neurons, termed hot neurons, are consistently activated across inputs, while the majority, cold neurons, vary based on specific inputs. PowerInfer exploits such an insight to design a GPU-CPU hybrid inference engine: hot-activated neurons are preloaded onto the GPU for fast access, while cold-activated neurons are computed on the CPU, thus significantly reducing GPU memory demands and CPU-GPU data transfers. PowerInfer further integrates adaptive predictors and neuron-aware sparse operators, optimizing the efficiency of neuron activation and computational sparsity. The evaluation shows that PowerInfer significantly outperforms llama.cpp by up to 11.69× while retaining model accuracy across various LLMs (including OPT-175B) on a single NVIDIA RTX 4090 GPU. For the OPT-30B model, PowerInfer achieves performance comparable to that of a high-end server-grade A100 GPU, reaching 82% of its token generation rate on a single consumer-grade RTX 4090 GPU.

1 September 2020 in China Communications
Verified
Journal Article
Closed access
ZHZheng Huang
Zheng Huang
ZMZeyu Mi
Zeyu Mi
ZHZhichao Hua
Zhichao Hua

30 March 2025
Verified
Conference Paper
Open access
YTYifan Tan
Yifan Tan
CTCheng Tan
Cheng Tan
ZMZeyu Mi
Zeyu Mi
HCHaibo Chen
Haibo Chen
Confidential computing on GPUs, like NVIDIA H100, mitigates the security risks of outsourced Large Language Models (LLMs) by implementing strong isolation and data encryption. Nonetheless, this encryption incurs a significant performance overhead, reaching up to 52.8% and 88.2% throughput drop when serving OPT-30B and OPT-66B, respectively. To address this challenge, we introduce PipeLLM, a user-transparent runtime system. PipeLLM removes the overhead by overlapping the encryption and GPU computation through pipelining-an idea inspired by the CPU instruction pipelining-thereby effectively concealing the latency increase caused by encryption. The primary technical challenge is that, unlike CPUs, the encryption module lacks prior knowledge of the specific data needing encryption until it is requested by the GPUs. To this end, we propose speculative pipelined encryption to predict the data requiring encryption by analyzing the serving patterns of LLMs. Further, we have developed an efficient, low-cost pipeline relinquishing approach for instances of incorrect predictions. Our experiments show that compared with vanilla systems without confidential computing (e.g., vLLM, PEFT, and FlexGen), PipeLLM incurs modest overhead (<19.6% in throughput) across various LLM sizes, from 13B to 175B. PipeLLM's source code is available at https://github.com/SJTU-IPADS/PipeLLM.

13 June 2018 in IEEE Transactions on Dependable and Secure Computing
Verified
Journal Article
Closed access
ZMZeyu Mi
Zeyu Mi
HCHaibo Chen
Haibo Chen
YZYinqian Zhang
Yinqian Zhang
SPShuanghe Peng
Shuanghe Peng
XWXiaofeng Wang
Xiaofeng Wang
MRMichael Reiter
Michael Reiter
In this paper, We present a new technique that offers lightweight, general, and elastic protection against Crum (Cross-VM runtime monitoring) attacks. Our protection, called CREASE (CPU Resource Elasticity as a Service), enables a VM (called principal) to purchase a higher clock rate from the cloud, through lowering the frequency of a malicious VM (called peer), to support its security-critical operations within a short period. During that period, the weakened peer becomes unable to catch up with the pace of the strengthened principal, therefore losing the capability to effectively collect its sensitive information. In the meantime, our approach can also make up for the performance impact on the peer through refunding schedule credits or service credits, in line with the service level agreement of today's cloud. At the center of our design is the novel application of on-demand frequency scaling and schedule quantum randomization, together with a situation-awareness mechanism that dynamically assesses the security risk posed by the peer. We analyzed the security guarantee of our design, implemented a prototype and evaluated it on a well-known Crum attack (an LLC side-channel attack) and various workloads. Our study shows that CREASE is effective at protecting the principal, with only a small impact on the peer's operations.

25 March 2023
Verified
Conference Paper
Open access
YLYuxuan Liu
Yuxuan Liu
TXTianqiang Xu
Tianqiang Xu
ZMZeyu Mi
Zeyu Mi
ZHZhichao Hua
Zhichao Hua
BZBinyu Zang
Binyu Zang
HCHaibo Chen
Haibo Chen
Today's cloud platforms offer large virtual machine (VM) instances with multiple virtual CPUs (vCPU) on manycore machines. These machines typically have a deep memory hierarchy to enhance communication between cores. Although previous researches have primarily focused on addressing the performance scalability issues caused by the double scheduling problem in virtualized environments, they mainly concentrated on solving the preemption problem of synchronization primitives and the traditional NUMA architecture. This paper specifically targets a new aspect of scalability issues caused by the absence of runtime hypervisor-internal states (RHS). We demonstrate two typical RHS problems, namely the invisible pCPU (physical CPU) load and dynamic cache group mapping. These RHS problems result in a collapse in VM performance and low CPU utilization because the guest VM lacks visibility into the latest runtime internal states maintained by the hypervisor, such as pCPU load and vCPU-pCPU mappings. Consequently, the guest VM makes inefficient scheduling decisions. To address the RHS issue, we argue that the solution lies in exposing the latest scheduling decisions made by both the guest and host schedulers to each other. Hence, we present a cooperative para-virtualized scheduling framework called CPS, which facilitates the proactive exchange of timely scheduling information between the hypervisor and guest VMs. To ensure effective scheduling decisions for VMs, a series of techniques are proposed based on the exchanged information. We have implemented CPS in Linux KVM and have designed corresponding solutions to tackle the two RHS problems. Evaluation results demonstrate that CPS significantly improves the performance of PARSEC by 81.1% and FxMark by 1.01x on average for the two identified problems.

25 November 2021 in IEEE Transactions on Computers
Verified
Journal Article
Closed access
ZMZeyu Mi
Zeyu Mi
HZHaoqi Zhuang
Haoqi Zhuang
BZBinyu Zang
Binyu Zang
HCHaibo Chen
Haibo Chen
IPC (Inter-Process Communication) is a widely used operating system (OS) technique that allows one process to invoke the services of other processes. The IPC participants may share the same OS ( internal IPC ) or use a separate OS ( external IPC ). Even though a long line of researches has optimized the performance of IPC, it is still a major factor of the run-time overhead of IPC-intensive applications. Furthermore, there is no one-size-fits-all solution for both internal and external IPC. This paper presents SkyBridge, a general communication technique designed and optimized for both types of IPC. SkyBridge requires no involvement of the privileged software (the kernel or the hypervisor) and enables a process to directly switch to the virtual address space of the target process, regardless of whether they are running on the same OS or not. We have implemented SkyBridge on two microkernels (seL4 and Google Zircon) as well as an open-source serverless hypervisor (Firecracker). The evaluation results show that SkyBridge improves the latency of internal IPC and external IPC by up to 19.6x and 1265.7x, respectively.