I’m co-organizing a workshop on Simulation for IR Evaluation at SIGIR this year. Below is an extract from the CfP:
Simulation techniques are not foreign to information retrieval. Simulation has been employed, for example, for constructing test collections and for model performance prediction and analysis in a broad array of information access scenarios. The need for simulation has become ever more apparent recently with the emergence of areas where other types of evaluation are infeasible. One such area is conversational information access, where human evaluation is both time and resource intensive at scale. Another example is provided by settings that do not allow sharing of data, e.g., because of privacy constraints, and therefore necessitate the creation of synthetic test collections.
Despite the apparent need, a standardized methodology for performance evaluation via simulation has not yet been developed. The goal of the Sim4IR workshop is to create a forum for researchers and practitioners to promote methodology development and more widespread use of simulation for evaluation by: (i) identifying problem settings and application scenarios; (ii) sharing tools, techniques, and experiences; (iii) characterizing potentials and limitations; and (iv) developing a research agenda.
Submission deadlines: May 4 (regular/short/demo papers) and May 18 (encore talks).
Visit sim4ir.org for more details.