Dialogue systems and large language models (LLMs) have gained considerable attention. However, the direct utilization of LLMs as task-oriented dialogue (TOD) models has been found to underperform compared to smaller task-specific models. Nonetheless, it is crucial to acknowledge the significant potential of LLMs and explore improved approaches for leveraging their impressive abilities. Motivated by the goal of leveraging LLMs, we propose an alternative approach called User-Guided Response Optimization (UGRO) to combine it with a smaller TOD model. This approach uses LLM as an annotation-free user simulator to assess dialogue responses, combining them with smaller fine-tuned end-to-end TOD models. By utilizing the satisfaction feedback generated by LLMs, UGRO further optimizes the supervised fine-tuned TOD model. Specifically, the TOD model takes the dialogue history as input and, with the assistance of the user simulator's feedback, generates high-satisfaction responses that meet the user's requirements. Through empirical experiments on two TOD benchmarks, we validate the effectiveness of our method. The results demonstrate that our approach outperforms previous state-of-the-art (SOTA) results.

Retrieval-augmented Generation (RAG) integrates Large Language Models (LLMs) with retrievers to access external knowledge, improving the factuality of LLM generation in knowledge-grounded tasks. To optimize the RAG performance, most previous work independently fine-tunes the retriever to adapt to frozen LLMs or trains the LLMs to use documents retrieved by off-the-shelf retrievers, lacking end-to-end training supervision. Recent work addresses this limitation by jointly training these two components but relies on overly simplifying assumptions of document independence, which has been criticized for being far from real-world scenarios. Thus, effectively optimizing the overall RAG performance remains a critical challenge. We propose a Direct Retrieval-augmented Optimization ( DRO ) framework that enables end-to-end training of two key components: (i) a generative knowledge selection model and (ii) an LLM generator. DRO alternates between two phases: (i) document permutation estimation and (ii) re-weighted maximization, progressively improving RAG components through a variational approach. In the estimation step, we treat document permutation as a latent variable and directly estimate its distribution from the selection model by applying an importance sampling strategy. In the maximization step, we calibrate the optimization expectation using importance weights and jointly train the selection model and LLM generator. Our theoretical analysis reveals that DRO is analogous to policy-gradient methods in reinforcement learning. Extensive experiments conducted on five datasets illustrate that DRO outperforms the best baseline with 5–15% improvements in EM and F1. We also qualitatively analyze the stability, convergence, and variance of DRO. (Code is available on DRO GitHub ).

As AI agents and tools increasingly carry out everyday tasks on behalf of users, the digital ecosystem is rapidly filling with agents that offer overlapping or complementary functionalities, raising a fundamental challenge: how can users, developers, and orchestrating systems effectively identify, compare, and select the most appropriate agents or tools for a given task? While rooted in traditional information retrieval (IR), agent search introduces new challenges because the retrieved entities are executable systems rather than static information artifacts, and their suitability depends on capabilities, behaviors, constraints, uncertainty, and task-dependent performance. We argue that the IR community is well-positioned to advance this emerging problem setting, which we refer to as AgentSearch, by developing principled methods for representing, indexing, retrieving, and ranking AI agents and tools under multi-dimensional relevance criteria. The AgentSearch Workshop aims to bring together researchers and practitioners from academia and industry to explore the challenges and opportunities of agent and tool search, including agent discovery, capability representation, retrieval and ranking models, evaluation methodologies, and issues related to safety, fairness, personalization, and explainability. To support concrete progress, the workshop will also host an exploratory AgentSearch Challenge that provides a shared experimental setting for ranking agents given a task. The workshop will adopt an interactive format featuring breakout discussions, poster and spotlight sessions, invited talks, and collaborative activities, fostering active engagement and cross-disciplinary exchange beyond a traditional mini-conference structure.