Longyue Wang, Chenyang Lyu, Tianbo Ji, Zhirui Zhang, Dian Yu, Shuming Shi, Zhaopeng Tu. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023.

Question Generation (QG) aims to automate the task of composing questions for a passage with a set of chosen answers found within the passage. In recent years, the introduction of neural generation models has resulted in substantial improvements of automatically generated questions in terms of quality, especially compared to traditional approaches that employ manually crafted heuristics. However, current QG evaluation metrics solely rely on the comparison between the generated questions and references, ignoring the passages or answers. Meanwhile, these metrics are generally criticized because of their low agreement with human judgement. We therefore propose a new reference-free evaluation metric called QAScore, which is capable of providing a better mechanism for evaluating QG systems. QAScore evaluates a question by computing the cross entropy according to the probability that the language model can correctly generate the masked words in the answer to that question. Compared to existing metrics such as BLEU and BERTScore, QAScore can obtain a stronger correlation with human judgement according to our human evaluation experiment, meaning that applying QAScore in the QG task benefits to a higher level of evaluation accuracy.

Large language models (LLMs) such as ChatGPT can produce coherent, cohesive, relevant, and fluent answers for various natural language processing (NLP) tasks. Taking document-level machine translation (MT) as a testbed, this paper provides an in-depth evaluation of LLMs' ability on discourse modeling. The study focuses on three aspects: 1) Effects of Context-Aware Prompts, where we investigate the impact of different prompts on document-level translation quality and discourse phenomena; 2) Comparison of Translation Models, where we compare the translation performance of ChatGPT with commercial MT systems and advanced document-level MT methods; 3) Analysis of Discourse Modelling Abilities, where we further probe discourse knowledge encoded in LLMs and shed light on impacts of training techniques on discourse modeling. By evaluating on a number of benchmarks, we surprisingly find that LLMs have demonstrated superior performance and show potential to become a new paradigm for document-level translation: 1) leveraging their powerful long-text modeling capabilities, GPT-3.5 and GPT-4 outperform commercial MT systems in terms of human evaluation; 2) GPT-4 demonstrates a stronger ability for probing linguistic knowledge than GPT-3.5. This work highlights the challenges and opportunities of LLMs for MT, which we hope can inspire the future design and evaluation of LLMs.We release our data and annotations at https://github.com/longyuewangdcu/Document-MT-LLM.

This study proposed an investigation-based multiple-criteria coordinated model to evaluate the sustainable development of urban public transport (PT) infrastructure, based on economic, social and environmental data from 2009 to 2019. The main problem with the traditional approach for assessing urban PT development is that economic and social benefits are considered individually, but also attention to environmental factors and coordination among the three issues are nearly overlooked. This leads to the likelihood of inaccuracies in the handling/assessment of sustainable development or an imbalance among the attributes in different cities. An investigation-based coordinated model was introduced in which a survey of 35 sub-criteria was conducted to derive the criteria necessary for coupling/coordination. A case study involving 13 cities in Jiangsu Province, China, illustrated the problems in coordinating PT systems and verified the efficacy of the proposed approach. With employing the entropy method, this study validated coordination of the PT infrastructure development of various cities in a balanced manner and used panel regression formulas to analyse the theoretical gap and empirical bottlenecks existing among economic, social and environmental benefits. With the findings of the study, the data-based investigation from 13 cities enabled the city planners/managers (including ones from other cities with similar urban levels) to give the individual priority between the ternary benefits, advance technology, allow big data-based informatisation and implement near-future autonomous PT vehicles.