In this work, we develop a specialized language model with strong discrete reasoning capabilities to tackle question answering (QA) over hybrid tabular and textual data in finance. Compared with adopting online LLMs, specializing smaller LLMs is more advantageous in response to users’ concerns about cost, network latency, and data security risks. To this end, we first abstract a Step-wise Pipeline for tabular and textual QA to help LLMs better execute multi-step inference, which includes three key steps, i.e. Extractor, Reasoner and Executor. This pipeline is proved to bring great performance grains compared with applying other prompting strategies like Chain-of-Thought (CoT), and meanwhile provides better interpretability to the derivation of the answer, with fixed inference steps and intermediate outcomes as references. We then develop a TAT-LLM model by fine-tuning LLaMA 2 with the training data generated automatically from existing datasets following the Step-wise Pipeline. The experimental results have verified that our TAT-LLM model can outperform all compared models, including prior best fine-tuned models and very large-scale LLMs like GPT-4 on FinQA, TAT-QA and TAT-DQA benchmarks. It is hoped that this work will shed light on practical solutions to the intelligent understanding of financial documents in the future. The generated datasets and trained models will be made publicly available to facilitate future research on the development of financial LLMs.

Retrieval-Augmented Generation (RAG) can alleviate hallucinations of Large Language Models (LLMs) by referencing external documents. However, the misinformation in external documents may mislead LLMs' generation. To address this issue, we explore the task of "credibility-aware RAG", in which LLMs automatically adjust the influence of retrieved documents based on their credibility scores to counteract misinformation. To this end, we introduce a plug-and-play method named Credibility-aware Attention Modification (CrAM). CrAM identifies influential attention heads in LLMs and adjusts their attention weights based on the credibility of the documents, thereby reducing the impact of low-credibility documents. Experiments on Natual Questions and TriviaQA using Llama2-13B, Llama3-8B, and Qwen1.5-7B show that CrAM improves the RAG performance of LLMs against misinformation pollution by over 20%, even surpassing supervised fine-tuning methods.

Towards the intelligent understanding of table-text data in the finance domain, previous research explores numerical reasoning over table-text content with Question Answering (QA) tasks. A general framework is to extract supporting evidence from the table and text and then perform numerical reasoning over extracted evidence for inferring the answer. However, existing models are vulnerable to missing supporting evidence, which limits their performance. In this work, we propose a novel Semantic-Oriented Hierarchical Graph (SoarGraph) that models the semantic relationships and dependencies among the different elements (e.g., question, table cells, text paragraphs, quantities, and dates) using hierarchical graphs to facilitate supporting evidence extraction and enhance numerical reasoning capability. We conduct our experiments on two popular benchmarks, FinQA and TAT-QA datasets, and the results show that our SoarGraph significantly outperforms all strong baselines, demonstrating remarkable effectiveness.

Yilun Qiu, Tianhao Shi, Xiaoyan Zhao, Fengbin Zhu, Yang Zhang, Fuli Feng. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 2025.