Evaluating the interactions between users and systems presents many challenges. Simulation offers a reliable, re-usable, and repeatable methodology to explore how different users, user behaviours and/or retrieval systems impact performance. With Large Language Models and Generative AI now widely available and accessible, new affordances are possible. These allow researchers to create more ''realistic'' simulated users that can generate queries and judge items like humans, and to develop new retrieval systems where responses and interactions are conversational and based on retrieval augmented generation. This resource paper presents a community-led initiative to update the Simulation of Interactive Information Retrieval (SimIIR) Framework to enable the simulation of conversational search using LLMs. The largest update provides a conversational search workflow which involves a number of new possible interactions with a search system or agent -- enabling a host of new development and evaluation opportunities. Other developments include the Markovian Users, Cognitive States, LLM-based components for assessing snippets/documents/responses, generating queries, deciding on when to stop/continue, and PyTerrier integration. This paper aims to mark the release of SimIIR 3.0 and invites the community to build, extend, and use the resource.
Search query suggestions affect users’ interactions with search engines, which then influences the information they encounter. Thus, bias in search query suggestions can lead to exposure to biased search results and can impact opinion formation. This is especially critical in the political domain. Detecting and quantifying bias in web search engines is difficult due to its topic dependency, complexity, and subjectivity. The lack of context and phrasality of query suggestions emphasizes this problem. In a multi-step approach, we combine the benefits of large language models, pairwise comparison, and Elo-based scoring to identify and quantify bias in English search query suggestions. We apply our approach to the U.S. political news domain and compare bias in Google and Bing.