Principled Multi-Aspect Evaluation Measures of Rankings

Publikation: Bidrag til bog/antologi/rapportKonferencebidrag i proceedingsForskningfagfællebedømt

Dokumenter

  • Fulltext

    Forlagets udgivne version, 1,4 MB, PDF-dokument

Information Retrieval evaluation has traditionally focused on defining principled ways of assessing the relevance of a ranked list of documents with respect to a query. Several methods extend this type of evaluation beyond relevance, making it possible to evaluate different aspects of a document ranking (e.g., relevance, usefulness, or credibility) using a single measure (multi-aspect evaluation). However, these methods either are (i) tailor-made for specific aspects and do not extend to other types or numbers of aspects, or (ii) have theoretical anomalies, e.g. assign maximum score to a ranking where all documents are labelled with the lowest grade with respect to all aspects (e.g., not relevant, not credible, etc.). We present a theoretically principled multi-aspect evaluation method that can be used for any number, and any type, of aspects. A thorough empirical evaluation using up to 5 aspects and a total of 425 runs officially submitted to 10 TREC tracks shows that our method is more discriminative than the state-of-the-art and overcomes theoretical limitations of the state-of-the-art.

OriginalsprogEngelsk
TitelCIKM 2021 - Proceedings of the 30th ACM International Conference on Information and Knowledge Management
ForlagAssociation for Computing Machinery, Inc
Publikationsdato2021
Sider1232-1242
ISBN (Elektronisk)9781450384469
DOI
StatusUdgivet - 2021
Begivenhed30th ACM International Conference on Information and Knowledge Management, CIKM 2021 - Virtual, Online, Australien
Varighed: 1 nov. 20215 nov. 2021

Konference

Konference30th ACM International Conference on Information and Knowledge Management, CIKM 2021
LandAustralien
ByVirtual, Online
Periode01/11/202105/11/2021
SponsorACM SIGIR, ACM SIGWEB

Bibliografisk note

Funding Information:
Acknowledgments. This paper is partially supported by the EU Horizon 2020 research and innovation programme under the MSCA grant No. 893667.

Publisher Copyright:
© 2021 Owner/Author.

Antal downloads er baseret på statistik fra Google Scholar og www.ku.dk


Ingen data tilgængelig

ID: 300918675