Injecting user models and time into precision via Markov chains

Publikation: Bidrag til bog/antologi/rapportKonferencebidrag i proceedingsForskningfagfællebedømt

We propose a family of new evaluation measures, called Markov Precision (MP), which exploits continuous-time and discrete-time Markov chains in order to inject user models into precision. Continuous-time MP behaves like timecalibrated measures, bringing the time spent by the user into the evaluation of a system; discrete-time MP behaves like traditional evaluation measures. Being part of the same Markovian framework, the time-based and rank-based versions of MP produce values that are directly comparable. We show that it is possible to re-create average precision using specific user models and this helps in providing an explanation of Average Precision (AP) in terms of user models more realistic than the ones currently used to justify it. We also propose several alternative models that take into account different possible behaviors in scanning a ranked result list. Finally, we conduct a thorough experimental evaluation of MP on standard TREC collections in order to show that MP is as reliable as other measures and we provide an example of calibration of its time parameters based on click logs from Yandex.

OriginalsprogEngelsk
TitelSIGIR 2014 - Proceedings of the 37th International ACM SIGIR Conference on Research and Development in Information Retrieval
Antal sider10
ForlagASSOCIATION FOR COMPUTING MACHINERY. JOU
Publikationsdato1 jan. 2014
Sider597-606
ISBN (Trykt)9781450322591
DOI
StatusUdgivet - 1 jan. 2014
Begivenhed37th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2014 - Gold Coast, QLD, Australien
Varighed: 6 jul. 201411 jul. 2014

Konference

Konference37th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2014
LandAustralien
ByGold Coast, QLD
Periode06/07/201411/07/2014
SponsorBaidu, et al., Google, Microsoft Research, Special Interest Group on Information Retrieval (ACM SIGIR), Tourism and Events Queensland

ID: 216517893