How to Robustly Combine Judgements from Crowd Assessors with AWARE

Publikation: Bidrag til tidsskrift › Konferenceartikel › Forskning › fagfællebedømt

Standard

How to Robustly Combine Judgements from Crowd Assessors with AWARE. / Ferrante, Marco; Ferro, Nicola; Maistro, Maria.

I: CEUR Workshop Proceedings, Bind 2161, 01.01.2018, s. 1DUMMY.

Publikation: Bidrag til tidsskrift › Konferenceartikel › Forskning › fagfællebedømt

Harvard

Ferrante, M, Ferro, N & Maistro, M 2018, 'How to Robustly Combine Judgements from Crowd Assessors with AWARE', CEUR Workshop Proceedings, bind 2161, s. 1DUMMY.

APA

Ferrante, M., Ferro, N., & Maistro, M. (2018). How to Robustly Combine Judgements from Crowd Assessors with AWARE. CEUR Workshop Proceedings, 2161, 1DUMMY.

Vancouver

Ferrante M, Ferro N, Maistro M. How to Robustly Combine Judgements from Crowd Assessors with AWARE. CEUR Workshop Proceedings. 2018 jan. 1;2161:1DUMMY.

Author

Ferrante, Marco ; Ferro, Nicola ; Maistro, Maria. / How to Robustly Combine Judgements from Crowd Assessors with AWARE. I: CEUR Workshop Proceedings. 2018 ; Bind 2161. s. 1DUMMY.

Bibtex

@inproceedings{82058290e7854e8093bb505be6d1b3d7,
title = "How to Robustly Combine Judgements from Crowd Assessors with AWARE",
abstract = "We propose the Assessor-driven Weighted Averages for Retrieval Evaluation (AWARE) probabilistic framework, a novel methodology for dealing with multiple crowd assessors, who may be contradictory and/or noisy. By modeling relevance judgements and crowd assessors as sources of uncertainty, AWARE directly combines the performance measures computed on the ground-truth generated by the crowd assessors instead of adopting some classification technique to merge the labels produced by them. We propose several unsupervised estimators that instantiate the AWARE framework and we compare them with Majority Vote (MV) and Expectation Maximization (EM) showing that AWARE approaches improve both in correctly ranking systems and predicting their actual performance scores.",
keywords = "AWARE, Crowdsourcing, Unsupervised estimators",
author = "Marco Ferrante and Nicola Ferro and Maria Maistro",
year = "2018",
month = jan,
day = "1",
language = "English",
volume = "2161",
pages = "1DUMMY",
journal = "CEUR Workshop Proceedings",
issn = "1613-0073",
publisher = "ceur workshop proceedings",
note = "26th Italian Symposium on Advanced Database Systems, SEBD 2018 ; Conference date: 24-06-2018 Through 27-06-2018",

}

RIS

TY - GEN

T1 - How to Robustly Combine Judgements from Crowd Assessors with AWARE

AU - Ferrante, Marco

AU - Ferro, Nicola

AU - Maistro, Maria

PY - 2018/1/1

Y1 - 2018/1/1

N2 - We propose the Assessor-driven Weighted Averages for Retrieval Evaluation (AWARE) probabilistic framework, a novel methodology for dealing with multiple crowd assessors, who may be contradictory and/or noisy. By modeling relevance judgements and crowd assessors as sources of uncertainty, AWARE directly combines the performance measures computed on the ground-truth generated by the crowd assessors instead of adopting some classification technique to merge the labels produced by them. We propose several unsupervised estimators that instantiate the AWARE framework and we compare them with Majority Vote (MV) and Expectation Maximization (EM) showing that AWARE approaches improve both in correctly ranking systems and predicting their actual performance scores.

AB - We propose the Assessor-driven Weighted Averages for Retrieval Evaluation (AWARE) probabilistic framework, a novel methodology for dealing with multiple crowd assessors, who may be contradictory and/or noisy. By modeling relevance judgements and crowd assessors as sources of uncertainty, AWARE directly combines the performance measures computed on the ground-truth generated by the crowd assessors instead of adopting some classification technique to merge the labels produced by them. We propose several unsupervised estimators that instantiate the AWARE framework and we compare them with Majority Vote (MV) and Expectation Maximization (EM) showing that AWARE approaches improve both in correctly ranking systems and predicting their actual performance scores.

KW - AWARE

KW - Crowdsourcing

KW - Unsupervised estimators

UR - http://www.scopus.com/inward/record.url?scp=85051853548&partnerID=8YFLogxK

M3 - Conference article

AN - SCOPUS:85051853548

VL - 2161

SP - 1DUMMY

JO - CEUR Workshop Proceedings

JF - CEUR Workshop Proceedings

SN - 1613-0073

T2 - 26th Italian Symposium on Advanced Database Systems, SEBD 2018

Y2 - 24 June 2018 through 27 June 2018

ER -

ID: 216516836