Chengtao Lai

Chengtao Lai

Auto-generated profile, based on publicly available information

Chengtao Lai

The Hong Kong University of Science and Technology

Research Focus & Metrics

Academic Career Length

3Years (2024-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
30 May 2024
Verified
Conference Paper
Open access
CLChengtao Lai
Chengtao Lai
ZZZhongchun Zhou
Zhongchun Zhou
APAkash Poptani
Akash Poptani
WZWei Zhang
Wei Zhang
The proliferation of large language models (LLMs) with substantial computational requirements and memory footprints has necessitated the design of more capable AI accelerators. Given the long compilation time of scratchpad memory-based (SPM-based) AI accelerators and the challenges brought by LLMs, we have explored the other side of the tradeoff - a multi-core AI accelerator system that incorporates a shared cache and application-specific management strategies - to provide significantly shorter compilation time at the cost of sometimes slightly lower performance than SPM-based systems. Besides, state-of-the-art mixed precision quantization methods also bring dynamic and irregular memory access patterns that do not fit SPMs well. Experiments conducted on cycle-accurate simulators have shown a maximum 30% memory time reduction with our cache management strategies, and up to 1.37x overall execution speedup with our prefetcher. Furthermore, we integrate the cache model into a state-of-the-art AI accelerator analytical model, allowing for a rapid understanding of the impact of caches for large-scale systems and big ML models. Overall, a system with our managed cache performs comparably with that using a conventional global SPM in non-quantized test-cases. However, our managed cache can outperform an SPM by 50.5% when a mixed precision quantization algorithm is used. With our cache management schemes, the proposed system can achieve a 2.57x speedup compared to LRU caches. Finally, we implement the design in RTL and the area of our design is 0.218mm2 with 15nm process, which can run at 1 GHz clock frequency. Our findings explore the potential of a shared cache design to assist the development of future AI accelerator systems.

4 September 2024 in ACM Transactions on Design Automation of Electronic Systems
Verified
Journal Article
Open access
CLChengtao Lai
Chengtao Lai
WZWei Zhang
Wei Zhang
Recent years have seen an increasing trend in designing AI accelerators together with the rest of the system, including CPUs and memory hierarchy. This trend calls for high-quality simulators or analytical models that enable such kind of co-exploration. Currently, the majority of such exploration is supported by AI accelerator analytical models. But such models usually overlook the non-trivial impact of congestion of shared resources, non-ideal hardware utilization and non-zero CPU scheduler overhead, which could only be modeled by cycle-level simulators. However, most simulators with full-stack toolchains are proprietary to corporations, and the few open-source simulators are suffering from either weak compilers or limited space of modeling. This framework resolves these issues by proposing a compilation and simulation flow to run arbitrary Caffe neural network models on the NVIDIA Deep Learning Accelerator (NVDLA) with gem5, a cycle-level simulator, and by adding more building blocks including scratchpad allocation, multi-accelerator scheduling, tensor-level prefetching mechanisms, and a DMA-aided embedded buffer to map workload to multiple NVDLAs. The proposed framework has been tested and verified on a set of convolution neural networks, showcasing the capability of modeling complex buffer management strategies, scheduling policies, and hardware architectures. As a case study of this framework, we demonstrate the importance of adopting different buffering strategies for activation and weight tensors in AI accelerators to acquire remarkable speedup.

8 December 2025 in arXiv
Verified
Preprint
Open access
PDF file
ZZZhongchun Zhou
Zhongchun Zhou
CLChengtao Lai
Chengtao Lai
YGYuhang Gu
Yuhang Gu
WZWei Zhang
Wei Zhang
The rapid adoption of large language models (LLMs) is pushing AI accelerators toward increasingly powerful and specialized designs. Instead of further complicating software development with deeply hierarchical scratchpad memories (SPMs) and their asynchronous management, we investigate the opposite point of the design spectrum: a multi-core AI accelerator equipped with a shared system-level cache and application-aware management policies, which keeps the programming effort modest. Our approach exploits dataflow information available in the software stack to guide cache replacement (including dead-block prediction), in concert with bypass decisions and mechanisms that alleviate cache thrashing. We assess the proposal using a cycle-accurate simulator and observe substantial performance gains (up to 1.80x speedup) compared with conventional cache architectures. In addition, we build and validate an analytical model that takes into account the actual overlapping behaviors to extend the measurement results of our policies to real-world larger-scale workloads. Experiment results show that when functioning together, our bypassing and thrashing mitigation strategies can handle scenarios both with and without inter-core data sharing and achieve remarkable speedups. Finally, we implement the design in RTL and the area of our design is $\mathbf{0.064mm^2}$ with 15nm process, which can run at 2 GHz clock frequency. Our findings explore the potential of the shared cache design to assist the development of future AI accelerator systems.


1 January 2026 in IEEE Transactions on Computers
Verified
Journal Article
Closed access
ZZZhongchun Zhou
Zhongchun Zhou
CLChengtao Lai
Chengtao Lai
YGYuhang Gu
Yuhang Gu
WZWei Zhang
Wei Zhang