Injecting user models and time into precision via Markov chains
Research output: Chapter in Book/Report/Conference proceeding › Article in proceedings › Research › peer-review
Standard
Injecting user models and time into precision via Markov chains. / Ferrante, Marco; Ferro, Nicola; Maistro, Maria.
SIGIR 2014 - Proceedings of the 37th International ACM SIGIR Conference on Research and Development in Information Retrieval. ASSOCIATION FOR COMPUTING MACHINERY. JOU, 2014. p. 597-606.Research output: Chapter in Book/Report/Conference proceeding › Article in proceedings › Research › peer-review
Harvard
APA
Vancouver
Author
Bibtex
}
RIS
TY - GEN
T1 - Injecting user models and time into precision via Markov chains
AU - Ferrante, Marco
AU - Ferro, Nicola
AU - Maistro, Maria
PY - 2014/1/1
Y1 - 2014/1/1
N2 - We propose a family of new evaluation measures, called Markov Precision (MP), which exploits continuous-time and discrete-time Markov chains in order to inject user models into precision. Continuous-time MP behaves like timecalibrated measures, bringing the time spent by the user into the evaluation of a system; discrete-time MP behaves like traditional evaluation measures. Being part of the same Markovian framework, the time-based and rank-based versions of MP produce values that are directly comparable. We show that it is possible to re-create average precision using specific user models and this helps in providing an explanation of Average Precision (AP) in terms of user models more realistic than the ones currently used to justify it. We also propose several alternative models that take into account different possible behaviors in scanning a ranked result list. Finally, we conduct a thorough experimental evaluation of MP on standard TREC collections in order to show that MP is as reliable as other measures and we provide an example of calibration of its time parameters based on click logs from Yandex.
AB - We propose a family of new evaluation measures, called Markov Precision (MP), which exploits continuous-time and discrete-time Markov chains in order to inject user models into precision. Continuous-time MP behaves like timecalibrated measures, bringing the time spent by the user into the evaluation of a system; discrete-time MP behaves like traditional evaluation measures. Being part of the same Markovian framework, the time-based and rank-based versions of MP produce values that are directly comparable. We show that it is possible to re-create average precision using specific user models and this helps in providing an explanation of Average Precision (AP) in terms of user models more realistic than the ones currently used to justify it. We also propose several alternative models that take into account different possible behaviors in scanning a ranked result list. Finally, we conduct a thorough experimental evaluation of MP on standard TREC collections in order to show that MP is as reliable as other measures and we provide an example of calibration of its time parameters based on click logs from Yandex.
KW - Evaluation
KW - Markov precision
KW - Time
KW - User model
UR - http://www.scopus.com/inward/record.url?scp=84904569307&partnerID=8YFLogxK
U2 - 10.1145/2600428.2609637
DO - 10.1145/2600428.2609637
M3 - Article in proceedings
AN - SCOPUS:84904569307
SN - 9781450322591
SP - 597
EP - 606
BT - SIGIR 2014 - Proceedings of the 37th International ACM SIGIR Conference on Research and Development in Information Retrieval
PB - ASSOCIATION FOR COMPUTING MACHINERY. JOU
T2 - 37th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2014
Y2 - 6 July 2014 through 11 July 2014
ER -
ID: 216517893