Publications

I’m excited to share that our EACL 2026 paper has been featured on the Google Research Blog!

We explore how to move beyond simple performance metrics to ensure simulated users actually behave like real ones and introduce a unique dual-agent data collection protocol that enables counterfactual validation. We also publicly release a new dataset of 4k+ human-AI shopping conversations.

Read the full deep-dive here: https://research.google/blog/convapparel-measuring-and-bridging-the-realism-gap-in-user-simulators/

I’m happy to share that our latest opinion piece, “The Indispensable Role of User Simulation in the Pursuit of AGI,” is now available in Communications of the ACM.

In this article, we argue that the path to Artificial General Intelligence (AGI) is currently blocked by two major bottlenecks: the lack of scalable evaluation and the scarcity of high-quality interaction data. We propose that user simulation is not just a helpful tool, but a critical catalyst for overcoming these challenges.

Read the full piece here: https://cacm.acm.org/opinion/the-indispensable-role-of-user-simulation-in-the-pursuit-of-agi/

I’m excited to share some recent research we’ve been doing in the areas of user simulation, recommender systems, and explainability. The following papers will be presented at the upcoming EACL and ECIR conferences. Importantly, all these papers come with publicly available resources!

• ConvApparel: A Benchmark Dataset and Validation Framework for User Simulators in Conversational Recommenders (EACL full paper, with Google colleagues O. Meshi, S. Goldman, A. Caciularu, G. Tennenholtz, J. Jeong, A. Globerson, and C. Boutilier) — This work proposes a comprehensive validation framework for user simulators, combining statistical alignment, a human-likeness score, and counterfactual validation.
• Trust Me on This: A User Study of Trustworthiness for RAG Responses (ECIR short paper, with W. Łajewska) — This study investigates how different types of explanations can influence user trust in a RAG setting.
• UserSimCRS v2: Simulation-Based Evaluation for Conversational Recommender Systems (ECIR resource paper, with N. Bernard) — This paper presents significant extensions to the UserSimCRS toolkit, including LLM-based simulators, support for a wider range of CRSs and datasets, and new evaluation metrics and utilities.
• Sim4IA-Bench: A User Simulation Benchmark Suite for Next Query and Utterance Prediction (ECIR resource paper, with A. K. Kruff, C. K. Kreutz, T. Breuer, and P. Schaer) — This work presents the simulation benchmark that is the result of the micro shared-tasks we ran at the Sim4IA workshop @SIGIR2025.
• SciNUP: Natural Language User Interest Profiles for Scientific Literature Recommendation (ECIR resource paper, with M. Arustashvili) — This paper introduces a synthetic dataset for NL profile-based recommendation in the scholarly domain.

I’m thrilled to announce that our book with ChengXiang Zhai “User Simulation for Evaluating Information Access Systems” has been published (a preprint is available on arXiv). This comprehensive monograph delves into the pivotal role of user simulation in assessing the effectiveness of information access systems, such as search engines, recommender systems, and conversational assistants. Addressing the intricate challenges of evaluating these systems, our book explores user simulation techniques that account for the diverse behaviours and preferences of users. It offers a detailed examination of general frameworks, models, and algorithms designed to simulate user interactions, and establishes connections with related fields, including machine learning, dialogue systems, user modeling, and economics. We also discuss future research directions that extend beyond the evaluation of information access systems and are expected to have broader impact on how to evaluate interactive intelligent systems in general.

I’m excited to share the draft of our book, co-authored by ChengXiang Zhai, User Simulation for Evaluating Information Access Systems: http://arxiv.org/abs/2306.08550

This book focuses on providing a thorough understanding of user simulation techniques designed specifically for evaluation purposes. We systematically review both general frameworks and specific models and algorithms for simulating user interactions with search engines, recommender systems, and conversational assistants.

We invite feedback on this first version. If you have suggestions, comments, pointers, etc. reach out to me in email!