Workshops

Living Labs for Information Retrieval Evaluation

Evaluation is a central aspect of information retrieval (IR) research. In the past few years, a new evaluation methodology known as living labs has been proposed as a way for researchers to be able to perform in-situ evaluation. This is not new, you might say; major web search engines have been doing it for serveral years already. While this is very true, it also means that this type of experimentation, with real users performing tasks using real-world applications, is only available to those selected few who are involved with the research labs of these organizations. There has been a lot of complaining about the “data divide” between industry and academia; living labs might be a way to bridge that.

The Living Labs for Information Retrieval Evaluation (LL’13) workshop at CIKM last year was a first attempt to bring people, both from academia and industry, together to discuss challenges and to formulate practical next steps. The workshop was successful in identifying and documenting possible further directions. See the preprint of the workshop summary.

The second edition of the iving Labs for IR workshop (LL’14), will run at CIKM this year. Our main goals are to continue our community building efforts around living labs for IR and to pursue the directions set out at LL’13. Having a community benchmarking platform with shared tasks would be a key catalyst in enabling people to make progress in this area. This is exactly what we are trying to set up for LL’14, in the form of a challenge (with the ultimate goal of turning it into a TREC, NTCIR or CLEF track in the future).

The challenge focuses on two specific use-cases: product search and local domain search. The basic idea is that participants receive a set of 100 frequent queries along with candidate results for these queries, and some general collection statistics. They are then expected to produce rankings for each query and to upload these rankings through an API. These rankings are evaluated online, on real users, and the results of these evaluations are made available to the participants, again, through an API.

In preparation for this challenge, we are organising a challenge workshop in Amsterdam on the 6th of June. The programme includes invited talks and a “hackathon.” We have a limited number of travel grants available (for those coming from outside The Netherlands and coming from academia) to cover travel and accommodation expenses. These are available on a “first come first served” basis (at most one per institute). If you would like to make use of this opportunity, please let us know as soon as possible.

More details may be found on our brand-new website: living-labs.net.

Call for Demos | Living Labs for IR workshop

The Living Labs for Information Retrieval Evaluation (LL’13) workshop at CIKM’13 invites researchers and practitioners to present their innovative prototypes or practical developments in a dedicated demo track. Demo submissions must be based on an implemented system that pursues one or more aspects relevant to the interest areas of the workshop.

Authors are strongly encouraged to target scenarios that are rooted in real-world applications. One way to think about this is by considering the following: as a company operating a website/service/application, what methods could allow various academic groups to experiment with specific components of this website/service/application?
In particular, we seek prototypes that define specific component(s) in the context of some website/service/application, and allow for the testing and evaluation of alternative methods for that component. One example is search within a specific vertical (such as product or travel search engine), but we encourage authors to think outside the (search) box.

All accepted demos will be evaluated and considered for the Best Demo Award.
The Best Demo Award winner will receive an award of 750 EUR, offered by the ‘Evaluating Information Access Systems’ (ELIAS) ESF Research Networking Programme. The award can be used to cover travel, accommodation or other expenses in relation to attending and/or demo’ing at LL’13.

The submission deadline for demos and for all other contributions is July 22 (extended).

Further details can be found on the workshop website.

Living Labs for IR workshop @CIKM

Together with Liadh Kelly, David Elsweiler, Evangelos Kanoulas, and Mark Smucker, I’m co-organising a workshop on Living Labs for IR Evaluation at CIKM this year.

The basic idea of living labs for IR is that rather than individual research groups independently developing experimental search infrastructures and gathering their own groups of test searchers for IR evaluations, a central and shared experimental environment is developed to facilitate the sharing of resources.

Living labs would offer huge benefits to the community, such as: availability of, potentially larger, cohorts of real users and their behaviours, e.g. querying behaviours, for experiment purposes; cross-comparability across research centres; and greater knowledge transfer between industry and academia, when industry partners are involved. The need for this methodology is further amplified by the increased reliance of IR approaches on proprietary data; living labs are a way to bridge the data divide between academia and industry.

There are many challenges to be overcome before the benefits associated with living labs for IR can be realised, including challenges associated with living labs architecture and design, hosting, maintenance, security, privacy, participant recruiting, and scenarios and tasks for use development.

This workshop aims to bring together for the first time people interested in progressing the living labs for IR evaluation methodology. An interactive forum for researchers to share ideas and initiate collaborations will be provided, with the explicit goal of determining means for progressing towards living labs for IR and formulating practical next steps for progression.

See the Call-for-Papers for more details.

As part of the workshop, we are considering organising a challenge in the e-commerce domain with the involvement of a medium-sized online retailer. The goal of this challenge would be to (i) allow academics to work with real users and data (esp. those who otherwise would have no access to such data) and (ii) to provide a starting point for the discussions at the workshop.

We will set up and run this challenge if there is sufficient interest in the community. We have made a poll to collect some initial feedback — please let us know what you think!

JIWES summary

The First Joint International Workshop on Entity-oriented and Semantic Search (JIWES) was held on Aug 16, 2012 in Portland, Oregon, USA, in conjunction with the 35th Annual International ACM SIGIR Conference (SIGIR 2012). The objective for the workshop was to bring together academic researchers and industry practitioners working on entity-oriented search to discuss tasks and challenges, and to uncover the next frontiers for academic research on the topic. The workshop program accommodated two invited talks, eight refereed papers divided into two technical paper sessions, and a group discussion.

In the forthcoming issue of SIGIR Forum we give a detailed summary of the workshop; the preprint of this article is available here. The workshop papers are available online in the ACM Digital library and at the workshop website. The latter also contains copies of the slides for most presentations.

JIWES@SIGIR’12 CfP

Call for Papers
1st Joint Intl. Workshop on Entity-oriented and Semantic Search (JIWES)
http://km.aifb.kit.edu/ws/jiwes2012/

WORKSHOP THEME
The workshop encompasses various tasks and approaches that go beyond the traditional bag-of-words paradigm and incorporate an explicit representation of the semantics behind information needs and relevant content. This kind of semantic search, based on concepts, entities and relations between them, has attracted attention both from industry and from the research community. The workshop aims to bring people from different communities (IR, SW, DB, NLP, HCI, etc.) and backgrounds (both academics and industry practitioners) together, to identify and discuss emerging trends, tasks and challenges. This joint workshop is a sequel of the Entity-oriented and Semantic Search Workshop series held at different conferences in previous years.

TOPICS
The workshop aims to gather all works that discuss entities along three dimensions: tasks, data and interaction. Tasks include entity search (search for entities or documents representing entities), relation search (search entities related to an entity), as well as more complex tasks (involving multiple entities—spatiotemporal relations inclusive—, involving multiple queries). In the data dimension, we consider (web/enterprise) documents (possibly annotated with entities/relations), LOD, as well as user generated content. The interaction dimension gives room for research into user interaction with entities, also considering how to display results, as well as whether to aggregate over multiple entities to construct entity profiles.

The workshop especially encourages submissions on the interface of IR and other disciplines, such as the Semantic Web, Databases, Computational Linguistics, Data Mining, Machine Learning, or Human Computer Interaction. Examples of topic of interest include (but are not limited to):

  • Data acquisition and processing (crawling, storage, and indexing)
  • Dealing with noisy, vague and incomplete data
  • Integration of data from multiple sources
  • Identification, resolution, and representation of entities (in documents and in queries)
  • Retrieval and ranking
  • Semantic query modeling (detecting, modeling, and understanding search intents)
  • Novel entity-oriented information access tasks
  • Interaction paradigms (natural language, keyword-based, and hybrid interfaces) and result representation
  • Test collections and evaluation methodology
  • Case studies and applications

We particularly encourage formal evaluation of approaches using previously established evaluation benchmarks.

SUBMISSION INFORMATION
We invite submissions of regular research papers (max. 6 pages), position papers (max. 3 pages), and demo descriptions (max. 3 pages). All submissions will be reviewed by at least two program committee members, and will be assessed based on their novelty, technical quality, potential impact, and clarity of writing. Selection uses a standard double blind procedure. All accepted papers will be published as part of the SIGIR workshop proceedings and will be indexed in the ACM Digital Library.

Please, submit in PDF format to:
http://www.easychair.org/conferences/?conf=jiwes2012
Using the ACM SIG Proceedings style (for LaTeX, use the “Option 2” style):
http://www.acm.org/sigs/publications/proceedings-templates

BEST CONTRIBUTION AWARD
The best contribution (paper/presentation) will receive an award sponsored by Yandex.

WORKSHOP FORMAT
The workshop will comprise of invited talks, oral presentations, and open-forum discussions.

IMPORTANT DATES

  • Submissions due: July 2, 2012 extended to July 9, 2012
  • Notification of acceptance: July 23, 2012
  • Camera-ready submission: Aug 1, 2012
  • Workshop date: Aug 16, 2012

ORGANIZING COMMITTEE

  • Krisztian Balog (NTNU, Norway)
  • David Carmel (IBM Research Haifa)
  • Arjen P. de Vries (CWI/TU Delft, The Netherlands)
  • Daniel M. Herzig (Karlsruhe Institute of Technology, Germany)
  • Peter Mika (Yahoo! Research, Barcelona)
  • Haggai Roitman (IBM Research Haifa)
  • Ralf Schenkel (Saarland University/MPII)
  • Pavel Serdyukov (Yandex, Russia)
  • Thanh Tran Duc (Karlsruhe Institute of Technology, Germany)

PROGRAM COMMITTEE
To be announced.

CONTACT
jiwes.workshop@gmail.com