Shengjie Liu

Shengjie Liu

Auto-generated profile, based on publicly available information

Shengjie Liu

North Carolina State University University of Illinois Urbana-Champaign Northwestern University North Carolina State University University of Illinois Urbana-Champaign North Carolina State University

Research Focus & Metrics

Academic Career Length

3Years (2024-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
29 March 2024 in arXiv
Verified
Preprint
Open access
PDF file
SLShengjie Liu
Shengjie Liu
JWJing Wu
Jing Wu
JBJingyuan Bao
Jingyuan Bao
WWWenyi Wang
Wenyi Wang
Naira HovakimyanNaira Hovakimyan
Naira Hovakimyan
CHChristopher G Healey
Christopher G Healey
This paper describes an investigation of the robustness of large language models (LLMs) for retrieval augmented generation (RAG)-based summarization tasks. While LLMs provide summarization capabilities, their performance in complex, real-world scenarios remains under-explored. Our first contribution is LogicSumm, an innovative evaluation framework incorporating realistic scenarios to assess LLM robustness during RAG-based summarization. Based on limitations identified by LogiSumm, we then developed SummRAG, a comprehensive system to create training dialogues and fine-tune a model to enhance robustness within LogicSumm's scenarios. SummRAG is an example of our goal of defining structured methods to test the capabilities of an LLM, rather than addressing issues in a one-off fashion. Experimental results confirm the power of SummRAG, showcasing improved logical coherence and summarization quality. Data, corresponding model weights, and Python code are available online.