Yunke Cai

Yunke Cai

Auto-generated profile, based on publicly available information

Yunke Cai

Bytedance Technology Co Ltd

Research Focus & Metrics

Academic Career Length

1Year (2026-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
13 October 2026 in ACM Transactions on Architecture and Code Optimization
Verified
Journal Article
Open access
TMTianyun Ma
Tianyun Ma
QWQian Wang
Qian Wang
SLShengwen Liang
Shengwen Liang
YCYunke Cai
Yunke Cai
ZNZiyuan Nan
Ziyuan Nan
TCTianshi Chen
Tianshi Chen
Deploying advanced large language models on edge devices, such as smartphones and robotics, is a growing trend that enhances user data privacy and network connectivity resilience while preserving intelligent capabilities. However, such a task exhibits single-batch computing with incredibly low arithmetic intensity, which poses significant challenges of huge memory footprint and bandwidth demands on limited edge resources. To address these issues, we introduce Cambricon-FlexLLM, a chiplet-based hybrid architecture with an NPU and a dedicated NAND flash chip to enable efficient on-device inference of 70B LLMs. Such a hybrid architecture utilizes both the high computing capability of NPU and the data capacity of the NAND flash chip, with the proposed hardware-tiling strategy that minimizes the data movement overhead between NPU and NAND flash chip. Specifically, the NAND flash chip, enhanced by our innovative in-flash computing and on-die ECC techniques, excels at performing precise lightweight on-die processing. Simultaneously, the NPU collaborates with the flash chip for matrix operations and handles special function computations beyond the flash’s on-die processing capabilities. Furthermore, to exploit the activation sparsity prevalent in modern LLMs, we propose a co-activation neuron-inspired weight-reordering algorithm and a sparsity-aware dynamic partitioning scheme. These optimizations significantly reduce the transfer of ineffective weight data. Experimental results demonstrate that Cambricon-FlexLLM achieves an inference speed of 3.44 tokens/s for 70B LLMs and 36.34 tokens/s for 7B LLMs, outperforming state-of-the-art flash-offloading frameworks by 22×–45×. Leveraging activation sparsity, Cambricon-FlexLLM achieves an additional 1.7× average speedup, ranging from 1.3× to 2.0× compared with dense inference. These results show a path toward local 70B-class decode for privacy-sensitive, offline, and fallback use cases on resource-constrained edge devices, while also highlighting the need to manage prefill latency, energy, thermal behavior, and shared-storage constraints.