Nick Craswell

Nick Craswell

Auto-generated profile, based on publicly available information

Nick Craswell

Microsoft, Bellevue, USA

Research Focus & Metrics

Academic Career Length

26Years (2001-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
10 July 2024
Verified
Conference Paper
Open access
PTPaul Thomas
Paul Thomas
SSSeth Spielman
Seth Spielman
NCNick Craswell
Nick Craswell
BMBhaskar Mitra
Bhaskar Mitra
Much of the evaluation and tuning of a search system relies on relevance labels---annotations that say whether a document is useful for a given search and searcher. Ideally these come from real searchers, but it is hard to collect this data at scale, so typical experiments rely on third-party labellers who may or may not produce accurate annotations. Label quality is managed with ongoing auditing, training, and monitoring. We discuss an alternative approach. We take careful feedback from real searchers and use this to select a large language model (LLM), and prompt, that agrees with this feedback; the LLM can then produce labels at scale. Our experiments show LLMs are as accurate as human labellers and as useful for finding the best systems and hardest queries. LLM performance varies with prompt features, but also varies unpredictably with simple paraphrases. This unpredictability reinforces the need for high-quality "gold" labels.

1 September 2001
Verified
Conference Paper
Open access
PDF file
NCNick Craswell
Nick Craswell
DHDavid Hawking
David Hawking
SRStephen Robertson
Stephen Robertson

10 July 2024
Verified
Conference Paper
Open access
HRHossein A. Rahmani
Hossein A. Rahmani
NCNick Craswell
Nick Craswell
EYEmine Yilmaz
Emine Yilmaz
BMBhaskar Mitra
Bhaskar Mitra
DCDaniel Campos
Daniel Campos
Constructing test collections in Information Retrieval (IR) is vital for evaluating search algorithms. Obtaining a diverse set of user queries for test collection construction can be challenging, and acquiring relevance judgments, which indicate the appropriateness of retrieved documents, is often costly and resource-intensive. Generating synthetic datasets using Large Language Models (LLMs) has recently gained significant attention in various applications. In information retrieval, while previous work exploited the capabilities of LLMs to generate synthetic queries or documents to augment training data and improve the performance of ranking models, using LLMs for constructing synthetic test collections is relatively unexplored. Previous studies demonstrate that LLMs have the potential to generate synthetic relevance judgments for use in the evaluation of information retrieval systems. In this paper, we comprehensively investigate whether it is possible to use LLMs to construct fully synthetic test collections by generating not only synthetic judgments but also synthetic queries. To qualify the efficacy of synthetic queries for examining system ordering, we analyze how these synthetic data are suitable for building a reliable and reusable test collection and the potential risks of bias such test collections may exhibit towards LLM-based models. Our comprehensive experiments indicate that test collections generated using LLMs can effectively and reliably evaluate system performance.

18 July 2025
Verified
Conference Paper
Open access
LDLaura Dietz
Laura Dietz
OZOleg Zendel
Oleg Zendel
PBPeter Bailey
Peter Bailey
CCCharles L. A. Clarke
Charles L. A. Clarke
ECEllese Cotterill
Ellese Cotterill
NCNick Craswell
Nick Craswell

13 July 2025
Verified
Conference Paper
Open access
RPRonak Pradeep
Ronak Pradeep
NTNandan Thakur
Nandan Thakur
SUShivani Upadhyay
Shivani Upadhyay
DCDaniel Campos
Daniel Campos
NCNick Craswell
Nick Craswell
JLJimmy Lin
Jimmy Lin

10 July 2024
Verified
Conference Paper
Open access
HRHossein A. Rahmani
Hossein A. Rahmani
CSClemencia Siro
Clemencia Siro
MAMohammad Aliannejadi
Mohammad Aliannejadi
NCNick Craswell
Nick Craswell
CCCharles L. A. Clarke
Charles L. A. Clarke
EYEmine Yilmaz
Emine Yilmaz
Large language models (LLMs) have demonstrated increasing task-solving abilities not present in smaller models. Utilizing the capabilities and responsibilities of LLMs for automated evaluation (LLM4Eval) has recently attracted considerable attention in multiple research communities. For instance, LLM4Eval models have been studied in the context of automated judgments, natural language generation, and retrieval augmented generation systems. We believe that the information retrieval community can significantly contribute to this growing research area by designing, implementing, analyzing, and evaluating various aspects of LLMs with applications to LLM4Eval tasks. The main goal of LLM4Eval workshop is to bring together researchers from industry and academia to discuss various aspects of LLMs for evaluation in information retrieval, including automated judgments, retrieval-augmented generation pipeline evaluation, altering human evaluation, robustness, and trustworthiness of LLMs for evaluation in addition to their impact on real-world applications. We also plan to run an automated judgment challenge prior to the workshop, where participants will be asked to generate labels for a given dataset while maximising correlation with human judgments. The format of the workshop is interactive, including roundtable and keynote sessions and tends to avoid the one-sided dialogue of a mini-conference.