In this paper, we survey the methods and concepts developed for the evaluation of dialogue systems. Evaluation, in and of itself, is a crucial part during the development process. Often, dialogue systems are evaluated by means of human evaluations and questionnaires. However, this tends to be very cost- and time-intensive. Thus, much work has been put into finding methods which allow a reduction in involvement of human labour. In this survey, we present the main concepts and methods. For this, we differentiate between the various classes of dialogue systems (task-oriented, conversational, and question-answering dialogue systems). We cover each class by introducing the main technologies developed for the dialogue systems and then present the evaluation methods regarding that class.
We examine the paradigm of data-centric artificial intelligence (DCAI) as a solution to the obstacles that small and medium-sized enterprises (SMEs) face in adopting AI. While the prevalent model-centric approach emphasizes collecting large amounts of data, SMEs often suffer from small datasets, data drift, and sparse ML knowledge, which hinders them from implementing AI. DCAI, on the other hand, emphasizes to systematically engineer the data used to build an AI system. Our contribution is to provide a concrete, transferable implementation of a DCAI development process geared towards industrial application, specffically in machining and manufacturing, and demonstrate how it enhances data quality by fostering collaboration between domain experts and ML engineers. This added value can place AI at the disposal of more SMEs. We provide the necessary background for practitioners to follow the rationale behind DCAI and successfully deploy the provided process template.
In this paper, we propose a classifier for predicting sentiments of Italian Twitter messages. This work builds upon a deep learning approach where we leverage large amounts of weakly labelled data to train a 2-layer convolutional neural network. To train our network we apply a form of multi-task training. Our system participated in the EvalItalia-2016 competition and outperformed all other approaches on the sentiment analysis task.
Claudio Paonessa, Yanick Schraner, Jan Deriu, Manuela Hürlimann, Manfred Vogel, Mark Cieliebak. Findings of the Association for Computational Linguistics: EMNLP 2023. 2023.
Janick Michot, Manuela Hürlimann, Jan Deriu, Luzia Sauer, Katsiaryna Mlynchyk, Mark Cieliebak. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024.