Retrieval-enhanced machine learning (REML) refers to the use of information retrieval methods to support reasoning and inference in machine learning tasks. Although relatively recent, these approaches can substantially improve model performance. This includes improved generalization, knowledge grounding, scalability, freshness, attribution, interpretability and on-device learning. To date, despite being influenced by work in the information retrieval community, REML research has predominantly been presented in natural language processing (NLP) conferences. Our tutorial addresses this disconnect by introducing core REML concepts and synthesizing the literature from various domains in machine learning (ML), including, but beyond NLP. What is unique to our approach is that we used consistent notations, to provide researchers with a unified and expandable framework. The tutorial will be presented in lecture format based on an existing manuscript, with supporting materials and a comprehensive reading list available at https://retrieval-enhanced-ml.github.io/SIGIR-AP2024-tutorial.

Modern information access ecosystems consist of mixtures of systems, such as retrieval systems and large language models, and increasingly rely on marketplaces to mediate access to models, tools, and data, making competition between systems inherent to deployment. In such settings, outcomes are shaped not only by benchmark quality but also by competitive pressure, including user switching, routing decisions, and operational constraints. Yet evaluation is still largely conducted on static benchmarks with accuracy-focused measures that assume systems operate in isolation. This mismatch makes it difficult to predict post-deployment success and obscures competitive effects such as early-adoption advantages and market dominance. We introduce Marketplace Evaluation, a simulation-based paradigm that evaluates information access systems as participants in a competitive marketplace. By simulating repeated interactions and evolving user and agent preferences, the framework enables longitudinal evaluation and marketplace-level metrics, such as retention and market share, that complement and can extend beyond traditional accuracy-based metrics. We formalize the framework and outline a research agenda, motivated by business and economics, around marketplace simulation, metrics, optimization, and adoption in evaluation campaigns like TREC.

As AI agents and tools increasingly carry out everyday tasks on behalf of users, the digital ecosystem is rapidly filling with agents that offer overlapping or complementary functionalities, raising a fundamental challenge: how can users, developers, and orchestrating systems effectively identify, compare, and select the most appropriate agents or tools for a given task? While rooted in traditional information retrieval (IR), agent search introduces new challenges because the retrieved entities are executable systems rather than static information artifacts, and their suitability depends on capabilities, behaviors, constraints, uncertainty, and task-dependent performance. We argue that the IR community is well-positioned to advance this emerging problem setting, which we refer to as AgentSearch, by developing principled methods for representing, indexing, retrieving, and ranking AI agents and tools under multi-dimensional relevance criteria. The AgentSearch Workshop aims to bring together researchers and practitioners from academia and industry to explore the challenges and opportunities of agent and tool search, including agent discovery, capability representation, retrieval and ranking models, evaluation methodologies, and issues related to safety, fairness, personalization, and explainability. To support concrete progress, the workshop will also host an exploratory AgentSearch Challenge that provides a shared experimental setting for ranking agents given a task. The workshop will adopt an interactive format featuring breakout discussions, poster and spotlight sessions, invited talks, and collaborative activities, fostering active engagement and cross-disciplinary exchange beyond a traditional mini-conference structure.