Renze Chen

Renze Chen

Auto-generated profile, based on publicly available information

Renze Chen

Peking University, Beijing, Beijing, CHINA

Research Focus & Metrics

Academic Career Length

6Years (2021-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
28 December 2021 in IEEE Transactions on Parallel and Distributed Systems
Verified
Journal Article
Open access
SZSize Zheng
Size Zheng
RCRenze Chen
Renze Chen
YJYicheng Jin
Yicheng Jin
AWAnjiang Wei
Anjiang Wei
BWBingyang Wu
Bingyang Wu
YLYun Liang
Yun Liang
Deep neural networks (DNNs) are increasingly deployed in various image recognition and natural language processing applications. The continuous demand for accuracy and high performance has led to innovations in DNN design and a proliferation of new operators. However, existing DNN training frameworks such as PyTorch and TensorFlow only support a limited range of operators and rely on hand-optimized libraries to provide efficient implementations for these operators. To evaluate novel neural networks with new operators, the programmers have to either replace the holistic new operators with existing operators or provide low-level implementations manually. Therefore, a critical requirement for DNN training frameworks is to provide high-performance implementations for the neural networks containing new operators automatically in the absence of efficient library support. In this article, we introduce NeoFlow, which is a flexible framework for enabling efficient compilation for high-performance DNN training. NeoFlow allows the programmers to directly write customized expressions as new operators to be mapped to graph representation and low-level implementations automatically, providing both high programming productivity and high performance. First, NeoFlow provides expression-based automatic differentiation to support customized model definitions with new operators. Then, NeoFlow proposes an efficient compilation system that partitions the neural network graph into subgraphs, explores optimized schedules, and generates high-performance libraries for subgraphs automatically. Finally, NeoFlow develops an efficient runtime system to combine the compilation and training as a whole by overlapping their execution. In the experiments, we examine the numerical accuracy and performance of NeoFlow. The results show that NeoFlow can achieve similar or even better performance at the operator and whole graph level for DNNs compared to deep learning frameworks. Especially, for novel networks training, the geometric mean speedups of NeoFlow to PyTorch, TensorFlow, and CuDNN are 3.16X, 2.43X, and 1.92X, respectively.

27 April 2024
Verified
Conference Paper
Open access
RCRenze Chen
Renze Chen
ZDZijian Ding
Zijian Ding
SZSize Zheng
Size Zheng
CZChengrui Zhang
Chengrui Zhang
JLJingwen Leng
Jingwen Leng
YLYun Liang
Yun Liang
Recently, memory consumption of Deep Neural Network (DNN) rapidly increases, mainly due to long lifetimes and large shapes of tensors. Graph scheduling has emerged as an effective memory optimization technique, which determines the optimal execution, re-computation, swap-out, and swap-in timings for each operator/tensor. However, it often hurts performance significantly and can only manipulate tensors' lifetimes but not shapes, limiting the optimization space. We find that graph transformation, which can change the tensor shapes and graph structure, creates a new trade-off space between memory and performance. Nevertheless, graph transformation are applied separately so far, with primary focus on optimizing performance and not memory. In this paper, we propose MAGIS, a DNN memory optimization framework that coordinates graph transformation with graph scheduling. MAGIS uses a hierarchical tree to represent Fission Transformation (F-Trans), a type of transformation which can effectively reduce tensor shapes in a sub-graph. To keep the complexity low, we build a light-weight search space based on graph structure analysis. MAGIS decomposes graph scheduling into graph transformation and re-ordering and designs an incremental scheduling algorithm to alleviate the scheduling overhead after each graph transformation step to efficiently coordinate them. Experimental results show that compared to state-of-the-art works, MAGIS only uses 15%~85% of their peak memory usage with the same latency1 constraint and obtains a better Pareto boundary in dual-objective optimization of memory and performance. Our code is now available at https://github.com/pku-liang/MAGIS.

23 June 2024
Verified
Conference Paper
Open access
RCRenze Chen
Renze Chen
ZDZijian Ding
Zijian Ding
SZSize Zheng
Size Zheng
MLMeng Li
Meng Li
YLYun Liang
Yun Liang
There has been a growing trend in deploying deep neural networks (DNNs) on tiny devices. However, deploying DNNs on such devices poses significant challenges due to the contradiction between DNNs' substantial memory requirements and the stringent memory constraints of tiny devices. Some prior works incur large latency overhead to save memory and target only simple CNNs, while others employ coarse-grained scheduling for complicated networks, leading to limited memory footprint reduction. This paper proposes MoteNN that performs fine-grained scheduling via operator partitioning on arbitrary DNNs to dramatically reduce peak memory usage with little latency overhead. MoteNN presents a graph representation named Axis Connecting Graph (ACG) to perform operator partition at graph-level efficiently. MoteNN further proposes an algorithm that finds the partition and schedule guided by memory bottlenecks. We evaluate MoteNN using various popular networks and show that MoteNN achieves up to 80% of peak memory usage reduction compared to the state-of-art works with nearly no latency overhead on tiny devices.

27 October 2024
Verified
Conference Paper
Open access
ZYZebin Yang
Zebin Yang
RCRenze Chen
Renze Chen
TWTaiqiang Wu
Taiqiang Wu
NWNgai Wong
Ngai Wong
YLYun Liang
Yun Liang
MLMeng Li
Meng Li

27 October 2024
Verified
Conference Paper
Open access
JYJiangrui Yu
Jiangrui Yu
WZWenxuan Zeng
Wenxuan Zeng
TXTianshi Xu
Tianshi Xu
RCRenze Chen
Renze Chen
YLYun Liang
Yun Liang
MLMeng Li
Meng Li

26 July 2026
Verified
Conference Paper
Open access
SLShanghao Liu
Shanghao Liu
RCRenze Chen
Renze Chen
SZSize Zheng
Size Zheng
YLYuanqiang Liu
Yuanqiang Liu
YLYun (Eric) Liang
Yun (Eric) Liang
HYHailong Yang
Hailong Yang