Shichun Liu

Shichun Liu

Auto-generated profile, based on publicly available information

Shichun Liu

Oil and Gas Technology Institute, PetroChina Changqing Oilfield Company Xi'an People's Republic of China

Research Focus & Metrics

Academic Career Length

3Years (2024-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
24 March 2024 in Proceedings of the AAAI Conference on Artificial Intelligence
Verified
Journal Article
Open access
PDF file
YZYue Zhang
Yue Zhang
MZMing Zhang
Ming Zhang
HYHaipeng Yuan
Haipeng Yuan
SLShichun Liu
Shichun Liu
YSYongyao Shi
Yongyao Shi
XHXuanjing Huang
Xuanjing Huang
Recently, the evaluation of Large Language Models has emerged as a popular area of research. The three crucial questions for LLM evaluation are ``what, where, and how to evaluate''. However, the existing research mainly focuses on the first two questions, which are basically what tasks to give the LLM during testing and what kind of knowledge it should deal with. As for the third question, which is about what standards to use, the types of evaluators, how to score, and how to rank, there hasn't been much discussion. In this paper, we analyze evaluation methods by comparing various criteria with both manual and automatic evaluation, utilizing onsite, crowd-sourcing, public annotators and GPT-4, with different scoring methods and ranking systems. We propose a new dataset, LLMEval and conduct evaluations on 20 LLMs. A total of 2,186 individuals participated, leading to the generation of 243,337 manual annotations and 57,511 automatic evaluation results. We perform comparisons and analyses of different settings and conduct 10 conclusions that can provide some insights for evaluating LLM in the future. The dataset and the results are publicly available at https://github.com/llmeval. The version with the appendix are publicly available at https://arxiv.org/abs/2312.07398.

12 August 2024 in Journal of Applied Polymer Science
Verified
Journal Article
Closed access
SLShichun Liu
Shichun Liu
WTWei Tian
Wei Tian
XWXianwen Wang
Xianwen Wang
XYXudong Yang
Xudong Yang
SXShuqin Xiao
Shuqin Xiao
MZMingwei Zhao
Mingwei Zhao