The University of Stavanger is one of the higher education institutions in Norway that are now working on a pilot project for digital examination. My web programming and interaction design course will be among the first ones (and the very first one within the Faculty of Science and Technology) that will have a digital exam already this semester.
The latest issue of Teknisk Ukeblad featured an article on this topic where I was also interviewed about my reasons for having a digital, as opposed to paper-based, exam.
The highlighed quote, in free translation, reads like “The point of the exam is that it should be as close to a work situation as possible. Students will not be doing programming on paper when they start their job.” Digital exams also allow for certain websites to be used as reference material, in addition to textbooks.
There have been a number of developments over the past months around our living labs for IR evaluation efforts.
There are many challenges associated with operationalizing a living labs benchmarking campaign. Chief of these are incorporating results from experimental search systems into live production systems, and obtaining sufficiently many impressions from relatively low traffic sites. We propose that frequent (head) queries can be used to generate result lists offline, which are then interleaved with results of the production system for live evaluation. The choice of head queries is critical because (1) it removes a harsh requirement of providing rankings in real-time for query requests and (2) it ensures that experimental systems receive enough impressions, on the same set of queries, for a meaningful comparison. This idea is described in detail in an upcoming CIKM’14 short paper: Head First: Living Labs for Ad-hoc Search Evaluation.
A sad, but newsworthy development was that our CIKM’14 workshop got cancelled. It was our plan to organize a living labs challenge as part of the workshop. That challenge cannot be run as originally planned. Now we have something much better.
Living Labs for IR Evaluation (LL4IR) will run as a Lab at CLEF 2015 along the tagline “Give us your ranking, we’ll have it clicked!” The first edition of the lab will focus on three specific use-cases: (1) product search (on an e-commerce site), (2) local domain search (on a university’s website), (3) web search (through a commercial web search engine). See futher details here.
About half a year ago I advertised a PhD position in Semantic Entity Search. There were no eligible candidates, so it has been converted to a 2-year Postdoc position.
There is good flexibility topic-wise—as long as it’s about entities and semantics
Please feel free to contact me with any questions or for further information.
Details and application instructions can be found here.
You will notice that there is a number of projects advertised. It’s a department-funded position, so “may the best applicant win” is the name of the game. Meaning, the strongest candidate will be offered the position, irrespective of the project chosen.
Starting date: from Sept 1, 2014.
Application deadline: June 22, 2014.
Evaluation is a central aspect of information retrieval (IR) research. In the past few years, a new evaluation methodology known as living labs has been proposed as a way for researchers to be able to perform in-situ evaluation. This is not new, you might say; major web search engines have been doing it for serveral years already. While this is very true, it also means that this type of experimentation, with real users performing tasks using real-world applications, is only available to those selected few who are involved with the research labs of these organizations. There has been a lot of complaining about the “data divide” between industry and academia; living labs might be a way to bridge that.
The Living Labs for Information Retrieval Evaluation (LL’13) workshop at CIKM last year was a first attempt to bring people, both from academia and industry, together to discuss challenges and to formulate practical next steps. The workshop was successful in identifying and documenting possible further directions. See the preprint of the workshop summary.
The second edition of the iving Labs for IR workshop (LL’14), will run at CIKM this year. Our main goals are to continue our community building efforts around living labs for IR and to pursue the directions set out at LL’13. Having a community benchmarking platform with shared tasks would be a key catalyst in enabling people to make progress in this area. This is exactly what we are trying to set up for LL’14, in the form of a challenge (with the ultimate goal of turning it into a TREC, NTCIR or CLEF track in the future).
The challenge focuses on two specific use-cases: product search and local domain search. The basic idea is that participants receive a set of 100 frequent queries along with candidate results for these queries, and some general collection statistics. They are then expected to produce rankings for each query and to upload these rankings through an API. These rankings are evaluated online, on real users, and the results of these evaluations are made available to the participants, again, through an API.
In preparation for this challenge, we are organising a challenge workshop in Amsterdam on the 6th of June. The programme includes invited talks and a “hackathon.” We have a limited number of travel grants available (for those coming from outside The Netherlands and coming from academia) to cover travel and accommodation expenses. These are available on a “first come first served” basis (at most one per institute). If you would like to make use of this opportunity, please let us know as soon as possible.
More details may be found on our brand-new website: living-labs.net.