Paul Thomas

Paul Thomas

Auto-generated profile, based on publicly available information

Paul Thomas

Microsoft, Adelaide, Australia

Research Focus & Metrics

Academic Career Length

11Years (2016-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
10 July 2024
Verified
Conference Paper
Open access
PTPaul Thomas
Paul Thomas
SSSeth Spielman
Seth Spielman
NCNick Craswell
Nick Craswell
BMBhaskar Mitra
Bhaskar Mitra
Much of the evaluation and tuning of a search system relies on relevance labels---annotations that say whether a document is useful for a given search and searcher. Ideally these come from real searchers, but it is hard to collect this data at scale, so typical experiments rely on third-party labellers who may or may not produce accurate annotations. Label quality is managed with ongoing auditing, training, and monitoring. We discuss an alternative approach. We take careful feedback from real searchers and use this to select a large language model (LLM), and prompt, that agrees with this feedback; the LLM can then produce labels at scale. Our experiments show LLMs are as accurate as human labellers and as useful for finding the best systems and hardest queries. LLM performance varies with prompt features, but also varies unpredictably with simple paraphrases. This unpredictability reinforces the need for high-quality "gold" labels.

18 July 2023
Verified
Conference Paper
Open access
MAMarwah Alaofi
Marwah Alaofi
LGLuke Gallagher
Luke Gallagher
MSMark Sanderson
Mark Sanderson
FSFalk Scholer
Falk Scholer
PTPaul Thomas
Paul Thomas

8 December 2024
Verified
Conference Paper
Open access
MAMarwah Alaofi
Marwah Alaofi
PTPaul Thomas
Paul Thomas
FSFalk Scholer
Falk Scholer
MSMark Sanderson
Mark Sanderson
Large Language Models (LLMs) are increasingly being used to assess the relevance of information objects. This work reports on experiments to study the labelling of short texts (i.e., passages) for relevance, using multiple open-source and proprietary LLMs. While the overall agreement of some LLMs with human judgements is comparable to human-to-human agreement measured in previous research, LLMs are more likely to label passages as relevant compared to human judges, indicating that LLM labels denoting non-relevance are more reliable than those indicating relevance. This observation prompts us to further examine cases where human judges and LLMs disagree, particularly when the human judge labels the passage as non-relevant and the LLM labels it as relevant. Results show a tendency for many LLMs to label passages that include the original query terms as relevant. We therefore conduct experiments to inject query words into random and irrelevant passages, not unlike the way we inserted the query 'best café near me' into this paper. The results demonstrate that LLMs are highly influenced by the presence of query words in the passages under assessment, even if the wider passage has no relevance to the query. This tendency of LLMs to be fooled by the mere presence of query words demonstrates a weakness in our current measures of LLM labelling: relying on overall agreement misses important patterns of failures. There is a real risk of bias in LLM-generated relevance labels and, therefore, a risk of bias in rankers trained on those labels. Additionally, we investigate the effects of deliberately manipulating LLMs by instructing them to label passages as relevant, similar to the instruction 'this paper is perfectly relevant' inserted above. We find that such manipulation influences the performance of some LLMs, highlighting the critical need to consider potential vulnerabilities when deploying LLMs in real-world applications.

10 July 2024
Verified
Conference Paper
Open access
HRHossein A. Rahmani
Hossein A. Rahmani
CSClemencia Siro
Clemencia Siro
MAMohammad Aliannejadi
Mohammad Aliannejadi
NCNick Craswell
Nick Craswell
CCCharles L. A. Clarke
Charles L. A. Clarke
EYEmine Yilmaz
Emine Yilmaz
Large language models (LLMs) have demonstrated increasing task-solving abilities not present in smaller models. Utilizing the capabilities and responsibilities of LLMs for automated evaluation (LLM4Eval) has recently attracted considerable attention in multiple research communities. For instance, LLM4Eval models have been studied in the context of automated judgments, natural language generation, and retrieval augmented generation systems. We believe that the information retrieval community can significantly contribute to this growing research area by designing, implementing, analyzing, and evaluating various aspects of LLMs with applications to LLM4Eval tasks. The main goal of LLM4Eval workshop is to bring together researchers from industry and academia to discuss various aspects of LLMs for evaluation in information retrieval, including automated judgments, retrieval-augmented generation pipeline evaluation, altering human evaluation, robustness, and trustworthiness of LLMs for evaluation in addition to their impact on real-world applications. We also plan to run an automated judgment challenge prior to the workshop, where participants will be asked to generate labels for a given dataset while maximising correlation with human judgments. The format of the workshop is interactive, including roundtable and keynote sessions and tends to avoid the one-sided dialogue of a mini-conference.

19 March 2023
Verified
Conference Paper
Open access
CSChirag Shah
Chirag Shah
RWRyen White
Ryen White
PTPaul Thomas
Paul Thomas
BMBhaskar Mitra
Bhaskar Mitra
SSShawon Sarkar
Shawon Sarkar
NBNicholas Belkin
Nicholas Belkin
The importance of tasks in information retrieval (IR) has been long argued for, addressed in different ways, often ignored, and frequently revisited. For decades, scholars made a case for the role that a user’s task plays in how and why that user engages in search and what a search system should do to assist. But for the most part, the IR community has been too focused on query processing and assuming a search task to be a collection of user queries, often ignoring if or how such an assumption addresses the users accomplishing their tasks. With emerging areas of conversational agents and proactive IR, understanding and addressing users’ tasks has become more important than ever before. In this paper, we provide various perspectives on where the state-of-the-art is with regard to tasks in IR, what are some of the bottlenecks in deriving and using task information, and how do we go forward from here. In addition to covering relevant literature, the paper provides a synthesis of historical and current perspectives on understanding, extracting, and addressing task-focused search. To ground ongoing and future research in this area, we present a new framing device for tasks using a tree-like structure and various moves on that structure that allow different interpretations and applications. Presented as a combination of synthesis of ideas and past works, proposals for future research, and our perspectives on technical, social, and ethical considerations, this paper is meant to help revitalize the interest and future work in task-based IR.

10 March 2024
Verified
Conference Paper
Open access
OZOleg Zendel
Oleg Zendel
JCJ. Shane Culpepper
J. Shane Culpepper
FSFalk Scholer
Falk Scholer
PTPaul Thomas
Paul Thomas
Large language models (LLMs) are capable of assessing document and query characteristics, including relevance, and are now being used for a variety of different classification labeling tasks as well. This study explores how to use LLMs to classify an information need, often represented as a user query. In particular, our goal is to classify the cognitive complexity of the search task for a given “backstory”. Using 180 TREC topics and backstories, we show that GPT-based LLMs agree with human experts as much as other human experts. We also show that batching and ordering can significantly impact the accuracy of GPT-3.5, but rarely alter the quality of GPT-4 predictions. This study provides insights into the efficacy of large language models for annotation tasks normally completed by humans, and offers recommendations for other similar applications.