Unbiased offline recommender evaluation for missing-not-at-random implicit feedback
Publikation: Bidrag til tidsskrift › Konferenceartikel › Forskning › fagfællebedømt
Implicit-feedback Recommenders (ImplicitRec) leverage positive only user-item interactions, such as clicks, to learn personalized user preferences. Recommenders are often evaluated and compared offline using datasets collected from online platforms. These platforms are subject to popularity bias (i.e., popular items are more likely to be presented and interacted with), and therefore logged ground truth data are Missing-Not-At-Random (MNAR). As a result, the widely used Average-Over-All (AOA) evaluator is biased toward accurately recommending trendy items. In this paper, we (a) investigate evaluation bias of AOA and (b) develop an unbiased and practical offline evaluator for implicit MNAR datasets using the Inverse-Propensity-Scoring (IPS) technique. Through extensive experiments using four real-world datasets and four widely used algorithms, we show that (a) popularity bias is widely manifested in item presentation and interaction; (b) evaluation bias due to MNAR data pervasively exists in most cases where AOA is used to evaluate ImplicitRec; and (c) the unbiased estimator significantly reduces the AOA evaluation bias by more than 30% in the Yahoo! music dataset in terms of the Mean Absolute Error (MAE).
Originalsprog | Engelsk |
---|---|
Tidsskrift | RecSys 2018 - 12th ACM Conference on Recommender Systems |
Sider (fra-til) | 279-287 |
Antal sider | 9 |
DOI | |
Status | Udgivet - 27 sep. 2018 |
Eksternt udgivet | Ja |
Begivenhed | 12th ACM Conference on Recommender Systems, RecSys 2018 - Vancouver, Canada Varighed: 2 okt. 2018 → 7 okt. 2018 |
Konference
Konference | 12th ACM Conference on Recommender Systems, RecSys 2018 |
---|---|
Land | Canada |
By | Vancouver |
Periode | 02/10/2018 → 07/10/2018 |
Sponsor | ACM Special Interest Group on Computer-Human Interaction (SIGCHI) |
Bibliografisk note
Publisher Copyright:
© 2018 Association for Computing Machinery.
ID: 301825505