I’m thrilled to announce that our book with ChengXiang Zhai “User Simulation for Evaluating Information Access Systems” has been published (a preprint is available on arXiv). This comprehensive monograph delves into the pivotal role of user simulation in assessing the effectiveness of information access systems, such as search engines, recommender systems, and conversational assistants. Addressing the intricate challenges of evaluating these systems, our book explores user simulation techniques that account for the diverse behaviours and preferences of users. It offers a detailed examination of general frameworks, models, and algorithms designed to simulate user interactions, and establishes connections with related fields, including machine learning, dialogue systems, user modeling, and economics. We also discuss future research directions that extend beyond the evaluation of information access systems and are expected to have broader impact on how to evaluate interactive intelligent systems in general.
Highlights from 2023
Another year has passed, and it’s now time for the usual annual highlights post. Here are some of the notable events and achievements from 2023:
- Completed the first draft of our book “User Simulation for Evaluating Information Access Systems” co-authored with ChengXiang Zhai, and published it on arXiv.
- Delivered a tutorial with ChengXiang Zhai on simulation-based evaluation at CIKM’23.
- Co-organized the second workshop on Measuring the Quality of Explanations in Recommender Systems, this time held at RecSys’23.
- Published several papers on conversational information access, including user simulation (WSDM’23), datasets (SIGIR’23a, SIGIR’23b, and CIKM’23), and reproducibility (ECIR’23).
- Contributed to research on recommender systems, focusing on the impact of explanations (CHI’23) and LLM-based recommendations (RecSys’23).
- Received the Best Resource Paper Award with Weronika Łajewska at CIKM’23.
- Welcomed a new faculty member, Petra Galuscakova, to the IAI group, marking another milestone in our growth.
Looking forward to what 2024 has in store!
User Simulation tutorial at AAAI’24 and WWW’24
Together with ChengXiang Zhai, we will be giving our user simulation tutorial at AAAI’24 and WWW’24, customized to the respective audiences. The AAAI’24 edition adopts a broader perspective of user simulation for evaluating an interactive AI system and focuses more on simulation algorithms and techniques that are well connected with various sub-fields of AI, such as machine learning and agent-based systems. In contrast, the WWW’24 edition emphasizes more on applications of user simulation for evaluating Web information access systems, including click modeling and the application of simulation techniques in e-commerce.
Tutorial on User Simulation @CIKM’23
I’m excited to share that, together with the esteemed ChengXiang Zhai, we’ll be presenting a tutorial on “User Simulation for Evaluating Information Access Systems” at the upcoming 32nd ACM International Conference on Information and Knowledge Management (CIKM ’23) in Birmingham, UK, in October. This half-day tutorial is based on our upcoming FnTIR book. We look forward to sharing our insights and also hope for feedback that could shape the final version of the book. Visit our companion website usersim.ai for more info.
User Simulation book draft available
I’m excited to share the draft of our book, co-authored by ChengXiang Zhai, User Simulation for Evaluating Information Access Systems: http://arxiv.org/abs/2306.08550
This book focuses on providing a thorough understanding of user simulation techniques designed specifically for evaluation purposes. We systematically review both general frameworks and specific models and algorithms for simulating user interactions with search engines, recommender systems, and conversational assistants.
We invite feedback on this first version. If you have suggestions, comments, pointers, etc. reach out to me in email!