TY - GEN

T1 - Towards a formal framework for utility-oriented measurements of retrieval effectiveness

AU - Ferrante, Marco

AU - Ferro, Nicola

AU - Maistro, Maria

PY - 2015/9/27

Y1 - 2015/9/27

N2 - In this paper we present a formal framework to define and study the properties of utility-oriented measurements of retrieval effectiveness, like AP, RBP, ERR and many other popular IR evaluation measures. The proposed framework is laid in the wake of the representational theory of measurement, which provides the foundations of the modern theory of measurement in both physical and social sciences, thus contributing to explicitly link IR evaluation to a broader context. The proposed framework is minimal, in the sense that it relies on just one axiom, from which other properties are derived. Finally, it contributes to a better understanding and a clear separation of what issues are due to the inherent problems in comparing systems in terms of retrieval effectiveness and what others are due to the expected numerical properties of a measurement.

AB - In this paper we present a formal framework to define and study the properties of utility-oriented measurements of retrieval effectiveness, like AP, RBP, ERR and many other popular IR evaluation measures. The proposed framework is laid in the wake of the representational theory of measurement, which provides the foundations of the modern theory of measurement in both physical and social sciences, thus contributing to explicitly link IR evaluation to a broader context. The proposed framework is minimal, in the sense that it relies on just one axiom, from which other properties are derived. Finally, it contributes to a better understanding and a clear separation of what issues are due to the inherent problems in comparing systems in terms of retrieval effectiveness and what others are due to the expected numerical properties of a measurement.

KW - Balancing index

KW - Omomorphism

KW - Replacement

KW - Representational theory of measurement

KW - Swap

UR - http://www.scopus.com/inward/record.url?scp=84964389739&partnerID=8YFLogxK

U2 - 10.1145/2808194.2809452

DO - 10.1145/2808194.2809452

M3 - Article in proceedings

AN - SCOPUS:84964389739

SP - 21

EP - 30

BT - ICTIR 2015 - Proceedings of the 2015 ACM SIGIR International Conference on the Theory of Information Retrieval

PB - Association for Computing Machinery, Inc.

T2 - 5th ACM SIGIR International Conference on the Theory of Information Retrieval, ICTIR 2015

Y2 - 27 September 2015 through 30 September 2015

ER -