Jiyu Guo

Jiyu Guo

Auto-generated profile, based on publicly available information

Jiyu Guo

Harbin Institute of Technology (Shenzhen)

Research Focus & Metrics

Academic Career Length

2Years (2025-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
22 June 2025 in Proceedings of the ACM on Software Engineering
Verified
Journal Article
Open access
RWRuiqi Wang
Ruiqi Wang
JGJiyu Guo
Jiyu Guo
CGCuiyun Gao
Cuiyun Gao
GFGuodong Fan
Guodong Fan
CCChun Yong Chong
Chun Yong Chong
XXXin Xia
Xin Xia
Recently, large language models (LLMs) have been deployed to tackle various software engineering (SE) tasks like code generation, significantly advancing the automation of SE tasks. However, assessing the quality of these LLM-generated code and text remains challenging. The commonly used Pass@k metric necessitates extensive unit tests and configured environments, demands a high labor cost, and is not suitable for evaluating LLM-generated text. Conventional metrics like BLEU, which measure only lexical rather than semantic similarity, have also come under scrutiny. In response, a new trend has emerged to employ LLMs for automated evaluation, known as LLM-as-a-judge. These LLM-as-a-judge methods are claimed to better mimic human assessment than conventional metrics without relying on high-quality reference answers. Nevertheless, their exact human alignment in SE tasks remains unexplored. In this paper, we empirically explore LLM-as-a-judge methods for evaluating SE tasks, focusing on their alignment with human judgments. We select seven LLM-as-a-judge methods that utilize general-purpose LLMs, alongside two LLMs specifically fine-tuned for evaluation. After generating and manually scoring LLM responses on three recent SE datasets of code translation, code generation, and code summarization, we then prompt these methods to evaluate each response. Finally, we compare the scores generated by these methods with human evaluation. The results indicate that output-based methods reach the highest Pearson correlation of 81.32 and 68.51 with human scores in code translation and generation, achieving near-human evaluation, noticeably outperforming ChrF++, one of the best conventional metrics, at 34.23 and 64.92. Such output-based methods prompt LLMs to output judgments directly, and exhibit more balanced score distributions that resemble human score patterns. Finally, we provide insights and implications, concluding that current state-of-the-art LLM-as-a-judge methods can potentially replace human evaluations in certain SE tasks.

28 May 2025 in arXiv
Verified
Preprint
Open access
PDF file
YHYiming Huang
Yiming Huang
JGJiyu Guo
Jiyu Guo
WMWenxin Mao
Wenxin Mao
CGCuiyun Gao
Cuiyun Gao
PHPeiyi Han
Peiyi Han
QLQing Ling
Qing Ling
Converting natural language (NL) questions into SQL queries, referred to as Text-to-SQL, has emerged as a pivotal technology for facilitating access to relational databases, especially for users without SQL knowledge. Recent progress in large language models (LLMs) has markedly propelled the field of natural language processing (NLP), opening new avenues to improve text-to-SQL systems. This study presents a systematic review of LLM-based text-to-SQL, focusing on four key aspects: (1) an analysis of the research trends in LLM-based text-to-SQL; (2) an in-depth analysis of existing LLM-based text-to-SQL techniques from diverse perspectives; (3) summarization of existing text-to-SQL datasets and evaluation metrics; and (4) discussion on potential obstacles and avenues for future exploration in this domain. This survey seeks to furnish researchers with an in-depth understanding of LLM-based text-to-SQL, sparking new innovations and advancements in this field.

1 November 2025
Verified
Book Chapter
Closed access
PDF file
YHYiming Huang
Yiming Huang
JGJiyu Guo
Jiyu Guo
JZJichuan Zeng
Jichuan Zeng
CGCuiyun Gao
Cuiyun Gao
PHPeiyi Han
Peiyi Han
CLChuanyi Liu
Chuanyi Liu