Shiyue Zhang

Shiyue Zhang

Auto-generated profile, based on publicly available information

Shiyue Zhang

UNC Chapel Hill

Research Focus & Metrics

Academic Career Length

10Years (2017-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
1 January 2023
Verified
Conference Paper
Open access
PDF file
SZShiyue Zhang
Shiyue Zhang
DWDavid Wan
David Wan
MBMohit Bansal
Mohit Bansal
Shiyue Zhang, David Wan, Mohit Bansal. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2023.

1 January 2017
Verified
Conference Paper
Open access
FYFeng Yang
Feng Yang
SZShiyue Zhang
Shiyue Zhang
AZAndi Zhang
Andi Zhang
DWDong Wang
Dong Wang
AAAndrew Karl Abel
Andrew Karl Abel

1 January 2023
Verified
Conference Paper
Open access
PDF file
DWDavid Wan
David Wan
SZShiyue Zhang
Shiyue Zhang
MBMohit Bansal
Mohit Bansal
David Wan, Shiyue Zhang, Mohit Bansal. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023.

2 December 2023 in arXiv
Verified
Preprint
Open access
PDF file
DTDerek Tam
Derek Tam
AMAnisha Mascarenhas
Anisha Mascarenhas
SZShiyue Zhang
Shiyue Zhang
SKSarah Kwan
Sarah Kwan
MBMohit Bansal
Mohit Bansal
CRColin Raffel
Colin Raffel
While large language models (LLMs) have proven to be effective on a large variety of tasks, they are also known to hallucinate information. To measure whether an LLM prefers factually consistent continuations of its input, we propose a new benchmark called FIB(Factual Inconsistency Benchmark) that focuses on the task of summarization. Specifically, our benchmark involves comparing the scores an LLM assigns to a factually consistent versus a factually inconsistent summary for an input news article. For factually consistent summaries, we use human-written reference summaries that we manually verify as factually consistent. To generate summaries that are factually inconsistent, we generate summaries from a suite of summarization models that we have manually annotated as factually inconsistent. A model's factual consistency is then measured according to its accuracy, i.e.\ the proportion of documents where it assigns a higher score to the factually consistent summary. To validate the usefulness of FIB, we evaluate 23 large language models ranging from 1B to 176B parameters from six different model families including BLOOM and OPT. We find that existing LLMs generally assign a higher score to factually consistent summaries than to factually inconsistent summaries. However, if the factually inconsistent summaries occur verbatim in the document, then LLMs assign a higher score to these factually inconsistent summaries than factually consistent summaries. We validate design choices in our benchmark including the scoring method and source of distractor summaries. Our code and benchmark data can be found at https://github.com/r-three/fib.

8 October 2020 in arXiv
Verified
Preprint
Open access
PDF file
PHPeter Hase
Peter Hase
SZShiyue Zhang
Shiyue Zhang
HXHarry Xie
Harry Xie
MBMohit Bansal
Mohit Bansal
Data collection for natural language (NL) understanding tasks has increasingly included human explanations alongside data points, allowing past works to introduce models that both perform a task and generate NL explanations for their outputs. Yet to date, model-generated explanations have been evaluated on the basis of surface-level similarities to human explanations, both through automatic metrics like BLEU and human evaluations. We argue that these evaluations are insufficient, since they fail to indicate whether explanations support actual model behavior (faithfulness), rather than simply match what a human would say (plausibility). In this work, we address the problem of evaluating explanations from the model simulatability perspective. Our contributions are as follows: (1) We introduce a leakage-adjusted simulatability (LAS) metric for evaluating NL explanations, which measures how well explanations help an observer predict a model's output, while controlling for how explanations can directly leak the output. We use a model as a proxy for a human observer, and validate this choice with two human subject experiments. (2) Using the CoS-E and e-SNLI datasets, we evaluate two existing generative graphical models and two new approaches; one rationalizing method we introduce achieves roughly human-level LAS scores. (3) Lastly, we frame explanation generation as a multi-agent game and optimize explanations for simulatability while penalizing label leakage, which can improve LAS scores. We provide code for the experiments in this paper at https://github.com/peterbhase/LAS-NL-Explanations

2 February 2023 in arXiv
Verified
Preprint
Open access
PDF file
SSSwarnadeep Saha
Swarnadeep Saha
SZShiyue Zhang
Shiyue Zhang
PHPeter Hase
Peter Hase
MBMohit Bansal
Mohit Bansal
Current abstractive summarization models either suffer from a lack of clear interpretability or provide incomplete rationales by only highlighting parts of the source document. To this end, we propose the Summarization Program (SP), an interpretable modular framework consisting of an (ordered) list of binary trees, each encoding the step-by-step generative process of an abstractive summary sentence from the source document. A Summarization Program contains one root node per summary sentence, and a distinct tree connects each summary sentence (root node) to the document sentences (leaf nodes) from which it is derived, with the connecting nodes containing intermediate generated sentences. Edges represent different modular operations involved in summarization such as sentence fusion, compression, and paraphrasing. We first propose an efficient best-first search method over neural modules, SP-Search that identifies SPs for human summaries by directly optimizing for ROUGE scores. Next, using these programs as automatic supervision, we propose seq2seq models that generate Summarization Programs, which are then executed to obtain final summaries. We demonstrate that SP-Search effectively represents the generative process behind human summaries using modules that are typically faithful to their intended behavior. We also conduct a simulation study to show that Summarization Programs improve the interpretability of summarization models by allowing humans to better simulate model reasoning. Summarization Programs constitute a promising step toward interpretable and modular abstractive summarization, a complex task previously addressed primarily through blackbox end-to-end neural systems. Supporting code available at https://github.com/swarnaHub/SummarizationPrograms