Krisztian Balog

The Living Labs for Information Retrieval Evaluation (LL’13) workshop at CIKM’13 invites researchers and practitioners to present their innovative prototypes or practical developments in a dedicated demo track. Demo submissions must be based on an implemented system that pursues one or more aspects relevant to the interest areas of the workshop.

Authors are strongly encouraged to target scenarios that are rooted in real-world applications. One way to think about this is by considering the following: as a company operating a website/service/application, what methods could allow various academic groups to experiment with specific components of this website/service/application?
In particular, we seek prototypes that define specific component(s) in the context of some website/service/application, and allow for the testing and evaluation of alternative methods for that component. One example is search within a specific vertical (such as product or travel search engine), but we encourage authors to think outside the (search) box.

All accepted demos will be evaluated and considered for the Best Demo Award.
The Best Demo Award winner will receive an award of 750 EUR, offered by the ‘Evaluating Information Access Systems’ (ELIAS) ESF Research Networking Programme. The award can be used to cover travel, accommodation or other expenses in relation to attending and/or demo’ing at LL’13.

The submission deadline for demos and for all other contributions is July 22 (extended).

Further details can be found on the workshop website.

Together with Liadh Kelly, David Elsweiler, Evangelos Kanoulas, and Mark Smucker, I’m co-organising a workshop on Living Labs for IR Evaluation at CIKM this year.

The basic idea of living labs for IR is that rather than individual research groups independently developing experimental search infrastructures and gathering their own groups of test searchers for IR evaluations, a central and shared experimental environment is developed to facilitate the sharing of resources.

Living labs would offer huge benefits to the community, such as: availability of, potentially larger, cohorts of real users and their behaviours, e.g. querying behaviours, for experiment purposes; cross-comparability across research centres; and greater knowledge transfer between industry and academia, when industry partners are involved. The need for this methodology is further amplified by the increased reliance of IR approaches on proprietary data; living labs are a way to bridge the data divide between academia and industry.

There are many challenges to be overcome before the benefits associated with living labs for IR can be realised, including challenges associated with living labs architecture and design, hosting, maintenance, security, privacy, participant recruiting, and scenarios and tasks for use development.

This workshop aims to bring together for the first time people interested in progressing the living labs for IR evaluation methodology. An interactive forum for researchers to share ideas and initiate collaborations will be provided, with the explicit goal of determining means for progressing towards living labs for IR and formulating practical next steps for progression.

See the Call-for-Papers for more details.

As part of the workshop, we are considering organising a challenge in the e-commerce domain with the involvement of a medium-sized online retailer. The goal of this challenge would be to (i) allow academics to work with real users and data (esp. those who otherwise would have no access to such data) and (ii) to provide a starting point for the discussions at the workshop.

We will set up and run this challenge if there is sufficient interest in the community. We have made a poll to collect some initial feedback — please let us know what you think!