Matthias Hagen

Matthias Hagen

Auto-generated profile, based on publicly available information

Matthias Hagen

Friedrich-Schiller-Universitat Jena, Jena, Germany

Research Focus & Metrics

Academic Career Length

20Years (2007-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
9 August 2023
Verified
Conference Paper
Open access
GFGuglielmo Faggioli
Guglielmo Faggioli
LDLaura Dietz
Laura Dietz
CCCharles L. A. Clarke
Charles L. A. Clarke
GDGianluca Demartini
Gianluca Demartini
MHMatthias Hagen
Matthias Hagen
HWHenning Wachsmuth
Henning Wachsmuth
When asked, large language models~(LLMs) like ChatGPT claim that they can assist with relevance judgments but it is not clear whether automated judgments can reliably be used in evaluations of retrieval systems. In this perspectives paper, we discuss possible ways for~LLMs to support relevance judgments along with concerns and issues that arise. We devise a human--machine collaboration spectrum that allows to categorize different relevance judgment strategies, based on how much humans rely on machines. For the extreme point of 'fully automated judgments', we further include a pilot experiment on whether LLM-based relevance judgments correlate with judgments from trained human assessors. We conclude the paper by providing opposing perspectives for and against the use of~LLMs for automatic relevance judgments, and a compromise perspective, informed by our analyses of the literature, our preliminary experimental evidence, and our experience as IR~researchers.

18 July 2023
Verified
Conference Paper
Open access
MFMaik Fröbe
Maik Fröbe
JRJan Heinrich Reimer
Jan Heinrich Reimer
SMSean MacAvaney
Sean MacAvaney
NDNiklas Deckers
Niklas Deckers
SRSimon Reich
Simon Reich
MPMartin Potthast
Martin Potthast
We integrate irdatasets, ir_measures, and PyTerrier with TIRA in the Information Retrieval Experiment Platform (TIREx) to promote more standardized, reproducible, scalable, and even blinded retrieval experiments. Standardization is achieved when a retrieval approach implements PyTerrier's interfaces and the input and output of an experiment are compatible with ir_datasets and ir_measures. However, none of this is a must for reproducibility and scalability, as TIRA can run any dockerized software locally or remotely in a cloud-native execution environment. Version control and caching ensure efficient (re)execution. TIRA allows for blind evaluation when an experiment runs on a remote server or cloud not under the control of the experimenter. The test data and ground truth are then hidden from public access, and the retrieval software has to process them in a sandbox that prevents data leaks. We currently host an instance of TIREx with 15 corpora (1.9~billion documents) on which 32 shared retrieval tasks are based. Using Docker images of 50~standard retrieval approaches, we automatically evaluated all approaches on all tasks (50 ⋅ 32 = 1,600 runs) in less than a week on a midsize cluster (1,620 cores and 24 GPUs). This instance of TIREx is open for submissions and will be integrated with the IR Anthology, as well as released open source.

10 July 2024
Verified
Conference Paper
Open access
LGLukas Gienapp
Lukas Gienapp
HSHarrisen Scells
Harrisen Scells
NDNiklas Deckers
Niklas Deckers
JBJanek Bevendorff
Janek Bevendorff
SWShuai Wang
Shuai Wang
MPMartin Potthast
Martin Potthast
Recent advances in large language models have enabled the development of viable generative retrieval systems. Instead of a traditional document ranking, generative retrieval systems often directly return a grounded generated text as a response to a query. Quantifying the utility of the textual responses is essential for appropriately evaluating such generative ad hoc retrieval. Yet, the established evaluation methodology for ranking-based ad hoc retrieval is not suited for the reliable and reproducible evaluation of generated responses. To lay a foundation for developing new evaluation methods for generative retrieval systems, we survey the relevant literature from the fields of information retrieval and natural language processing, identify search tasks and system architectures in generative retrieval, develop a new user model, and study its operationalization.

1 May 2019 in Information Processing & Management
Verified
Journal Article
Closed access
PVPertti Vakkari
Pertti Vakkari
MVMichael Völske
Michael Völske
MPMartin Potthast
Martin Potthast
MHMatthias Hagen
Matthias Hagen
BSBenno Stein
Benno Stein

10 March 2024
Verified
Conference Paper
Open access
IZInes Zelch
Ines Zelch
MHMatthias Hagen
Matthias Hagen
MPMartin Potthast
Martin Potthast
Commercial conversational search engines need a business model. Since advertising is the main source of revenue for “traditional” ten-blue-links web search, ads are not an unlikely option for conversational search either. In traditional web search, ads are usually placed above organic search results. However, large language models (LLMs) may be dynamically prompted to blend product placements with “organic” conversational responses, similar to native advertising in journalism. This type of advertising can be very difficult to recognize, depending on how subtly it is integrated and disclosed. To raise awareness of this potential development, we analyze the capabilities of current LLMs to blend ads with generative search results. In a user study, we ask people about the perceived quality of (emulated) search results in different advertising scenarios. In a substantial number of cases, our survey participants do not notice brand or product placements when they do not expect them. Thus, our results show the potential of LLMs to subtly mix advertising with generated search results. This warrants further investigation, for example, to develop appropriate advertising disclosure rules, and to detect advertising in generated results. Our research also raises broader concerns about whether commercial or open-source generative models can be trusted not to be fine-tuned to generate ads rather than “genuine” responses.

4 April 2025
Verified
Book Chapter
Closed access
PDF file
FSFerdinand Schlatt
Ferdinand Schlatt
MFMaik Fröbe
Maik Fröbe
HSHarrisen Scells
Harrisen Scells
SZShengyao Zhuang
Shengyao Zhuang
BKBevan Koopman
Bevan Koopman
MHMatthias Hagen
Matthias Hagen