Tianqi Shao

Tianqi Shao

Auto-generated profile, based on publicly available information

Tianqi Shao

Peking University, Beijing, China

Research Focus & Metrics

Academic Career Length

1Year (2026-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
30 June 2026 in Proceedings of the ACM on Software Engineering
Verified
Journal Article
Open access
JLJia Li
Jia Li
HDHongyi Deng
Hongyi Deng
YZYiran Zhang
Yiran Zhang
KZKechi Zhang
Kechi Zhang
TSTianqi Shao
Tianqi Shao
YDYihong Dong
Yihong Dong
Writing code requires significant time and effort in software development. To automate this process, researchers have made substantial progress using Large Language Models (LLMs) for code generation. Many benchmarks like HumanEval and EvoCodeBench have been created to evaluate LLMs by requiring them to generate code from natural language requirements. However, in enterprise applications and team development, developers typically write code based on structured designs or specifications rather than raw natural language descriptions. This gap between existing benchmarks and real industry development practices means that current benchmark scores may not accurately reflect how much code generation can help automate software development tasks. To address this gap, we propose RealBench, a repository-level code generation benchmark aligned with real-world industry software development practices. Each example includes both natural language requirements and UML diagrams as system design, matching how developers typically receive specifications. Based on the constructed benchmarks, we conduct a systematic evaluation of advanced LLMs' code generation capabilities when provided with structured system designs. Specifically, we design three generation strategies to evaluate advanced LLMs on RealBench and propose two evaluation granularities with five metrics. The experimental results reveal key insights in current LLMs' capabilities for repo-level code generation aligned with real-world software development practices. First, we notice that regarding repo-level code generation, LLMs show much worse performance and there are significant performance gaps among LLMs. Second, LLMs are good at finding and creating modules defined in UML diagrams, but the quality of generated modules is often poor due to grammar and logic errors. Third, generating the entire repository at once is the best generation strategy on smaller repositories, while generating a complex repository with the module-by-module strategy works better compared to other strategies. Fourth, the detailed system design is very important for repository-level code generation tasks through conducting ablation studies on system designs. Lastly, we discuss the frequent error types in generated repositories to provide insights for optimizing repo-level code generation.