Ansatte – Københavns Universitet

An improved multileaving algorithm for online ranker evaluation

Publikation: Bidrag til bog/antologi/rapportKonferencebidrag i proceedingsForskningfagfællebedømt

Online ranker evaluation is a key challenge in information
retrieval. An important task in the online evaluation of
rankers is using implicit user feedback for inferring preferences between rankers. Interleaving methods have been
found to be ecient and sensitive, i.e. they can quickly detect even small dierences in quality. It has recently been
shown that multileaving methods exhibit similar sensitivity
but can be more ecient than interleaving methods. This
paper presents empirical results demonstrating that existing multileaving methods either do not scale well with the
number of rankers, or, more problematically, can produce
results which substantially dier from evaluation measures
like NDCG. The latter problem is caused by the fact that
they do not correctly account for the similarities that can
occur between rankers being multileaved. We propose a new
multileaving method for handling this problem and demonstrate that it substantially outperforms existing methods, in
some cases reducing errors by as much as 50%.
OriginalsprogEngelsk
TitelProceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval : SIGIR '16
Antal sider4
ForlagAssociation for Computing Machinery
Publikationsdato2016
Sider745-748
ISBN (Trykt)978-1-4503-4069-4
DOI
StatusUdgivet - 2016
BegivenhedInternational ACM SIGIR conference on Research and Development in Information Retrieval 2016: SIGIR '16 - Pisa, Italien
Varighed: 17 jul. 201621 jul. 2016
Konferencens nummer: 39
http://sigir.org/sigir2016/

Konference

KonferenceInternational ACM SIGIR conference on Research and Development in Information Retrieval 2016
Nummer39
LandItalien
ByPisa
Periode17/07/201621/07/2016
Internetadresse

ID: 164440333