Benno Stein

Benno Stein

Auto-generated profile, based on publicly available information

Benno Stein

Bauhaus-Universität Weimar, Weimar, Germany

Research Focus & Metrics

Academic Career Length

18Years (2009-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
1 March 2015 in International Journal of Artificial Intelligence in Education
Verified
Journal Article
Open access
PDF file
SBSteven Burrows
Steven Burrows
IGIryna Gurevych
Iryna Gurevych
BSBenno Stein
Benno Stein
Automatic short answer grading (ASAG) is the task of assessing short natural language responses to objective questions using computational methods. The active research in this field has increased enormously of late with over 80 papers fitting a definition of ASAG. However, the past efforts have generally been ad-hoc and non-comparable until recently, hence the need for a unified view of the whole field. The goal of this paper is to address this aim with a comprehensive review of ASAG research and systems according to history and components. Our historical analysis identifies 35 ASAG systems within 5 temporal themes that mark advancement in methodology or evaluation. In contrast, our component analysis reviews 6 common dimensions from preprocessing to effectiveness. A key conclusion is that an era of evaluation is the newest trend in ASAG research, which is paving the way for the consolidation of the field.

30 January 2010 in Language Resources and Evaluation
Verified
Journal Article
Closed access
PDF file
MPMartin Potthast
Martin Potthast
ABAlberto Barrón-Cedeño
Alberto Barrón-Cedeño
BSBenno Stein
Benno Stein
PRPaolo Rosso
Paolo Rosso

1 January 2016
Verified
Book Chapter
Closed access
PDF file
MPMartin Potthast
Martin Potthast
SKSebastian Köpsel
Sebastian Köpsel
BSBenno Stein
Benno Stein
MHMatthias Hagen
Matthias Hagen

20 January 2010 in Language Resources and Evaluation
Verified
Journal Article
Closed access
PDF file
BSBenno Stein
Benno Stein
NLNedim Lipka
Nedim Lipka
PPPeter Prettenhofer
Peter Prettenhofer

18 July 2023
Verified
Conference Paper
Open access
MFMaik Fröbe
Maik Fröbe
JRJan Heinrich Reimer
Jan Heinrich Reimer
SMSean MacAvaney
Sean MacAvaney
NDNiklas Deckers
Niklas Deckers
SRSimon Reich
Simon Reich
MPMartin Potthast
Martin Potthast
We integrate irdatasets, ir_measures, and PyTerrier with TIRA in the Information Retrieval Experiment Platform (TIREx) to promote more standardized, reproducible, scalable, and even blinded retrieval experiments. Standardization is achieved when a retrieval approach implements PyTerrier's interfaces and the input and output of an experiment are compatible with ir_datasets and ir_measures. However, none of this is a must for reproducibility and scalability, as TIRA can run any dockerized software locally or remotely in a cloud-native execution environment. Version control and caching ensure efficient (re)execution. TIRA allows for blind evaluation when an experiment runs on a remote server or cloud not under the control of the experimenter. The test data and ground truth are then hidden from public access, and the retrieval software has to process them in a sandbox that prevents data leaks. We currently host an instance of TIREx with 15 corpora (1.9~billion documents) on which 32 shared retrieval tasks are based. Using Docker images of 50~standard retrieval approaches, we automatically evaluated all approaches on all tasks (50 ⋅ 32 = 1,600 runs) in less than a week on a midsize cluster (1,620 cores and 24 GPUs). This instance of TIREx is open for submissions and will be integrated with the IR Anthology, as well as released open source.

10 July 2024
Verified
Conference Paper
Open access
LGLukas Gienapp
Lukas Gienapp
HSHarrisen Scells
Harrisen Scells
NDNiklas Deckers
Niklas Deckers
JBJanek Bevendorff
Janek Bevendorff
SWShuai Wang
Shuai Wang
MPMartin Potthast
Martin Potthast
Recent advances in large language models have enabled the development of viable generative retrieval systems. Instead of a traditional document ranking, generative retrieval systems often directly return a grounded generated text as a response to a query. Quantifying the utility of the textual responses is essential for appropriately evaluating such generative ad hoc retrieval. Yet, the established evaluation methodology for ranking-based ad hoc retrieval is not suited for the reliable and reproducible evaluation of generated responses. To lay a foundation for developing new evaluation methods for generative retrieval systems, we survey the relevant literature from the fields of information retrieval and natural language processing, identify search tasks and system architectures in generative retrieval, develop a new user model, and study its operationalization.