The multidimensional wisdom of crowds

Publikation: Bidrag til tidsskrift › Konferenceartikel › Forskning › fagfællebedømt

Standard

The multidimensional wisdom of crowds. / Welinder, Peter; Branson, Steve; Belongie, Serge; Perona, Pietro.

I: Advances in Neural Information Processing Systems 23: 24th Annual Conference on Neural Information Processing Systems 2010, NIPS 2010, 2010.

Publikation: Bidrag til tidsskrift › Konferenceartikel › Forskning › fagfællebedømt

Harvard

Welinder, P, Branson, S, Belongie, S & Perona, P 2010, 'The multidimensional wisdom of crowds', Advances in Neural Information Processing Systems 23: 24th Annual Conference on Neural Information Processing Systems 2010, NIPS 2010.

APA

Welinder, P., Branson, S., Belongie, S., & Perona, P. (2010). The multidimensional wisdom of crowds. Advances in Neural Information Processing Systems 23: 24th Annual Conference on Neural Information Processing Systems 2010, NIPS 2010.

Vancouver

Welinder P, Branson S, Belongie S, Perona P. The multidimensional wisdom of crowds. Advances in Neural Information Processing Systems 23: 24th Annual Conference on Neural Information Processing Systems 2010, NIPS 2010. 2010.

Author

Welinder, Peter ; Branson, Steve ; Belongie, Serge ; Perona, Pietro. / The multidimensional wisdom of crowds. I: Advances in Neural Information Processing Systems 23: 24th Annual Conference on Neural Information Processing Systems 2010, NIPS 2010. 2010.

Bibtex

@inproceedings{bc19882b28ac43c5b19fbeea36a66fe4,
title = "The multidimensional wisdom of crowds",
abstract = "Distributing labeling tasks among hundreds or thousands of annotators is an increasingly important method for annotating large datasets. We present a method for estimating the underlying value (e.g. the class) of each image from (noisy) annotations provided by multiple annotators. Our method is based on a model of the image formation and annotation process. Each image has different characteristics that are represented in an abstract Euclidean space. Each annotator is modeled as a multidimensional entity with variables representing competence, expertise and bias. This allows the model to discover and represent groups of annotators that have different sets of skills and knowledge, as well as groups of images that differ qualitatively. We find that our model predicts ground truth labels on both synthetic and real data more accurately than state of the art methods. Experiments also show that our model, starting from a set of binary labels, may discover rich information, such as different {"}schools of thought{"} amongst the annotators, and can group together images belonging to separate categories.",
author = "Peter Welinder and Steve Branson and Serge Belongie and Pietro Perona",
year = "2010",
language = "English",
journal = "Advances in Neural Information Processing Systems 23: 24th Annual Conference on Neural Information Processing Systems 2010, NIPS 2010",
note = "24th Annual Conference on Neural Information Processing Systems 2010, NIPS 2010 ; Conference date: 06-12-2010 Through 09-12-2010",

}

RIS

TY - GEN

T1 - The multidimensional wisdom of crowds

AU - Welinder, Peter

AU - Branson, Steve

AU - Belongie, Serge

AU - Perona, Pietro

PY - 2010

Y1 - 2010

N2 - Distributing labeling tasks among hundreds or thousands of annotators is an increasingly important method for annotating large datasets. We present a method for estimating the underlying value (e.g. the class) of each image from (noisy) annotations provided by multiple annotators. Our method is based on a model of the image formation and annotation process. Each image has different characteristics that are represented in an abstract Euclidean space. Each annotator is modeled as a multidimensional entity with variables representing competence, expertise and bias. This allows the model to discover and represent groups of annotators that have different sets of skills and knowledge, as well as groups of images that differ qualitatively. We find that our model predicts ground truth labels on both synthetic and real data more accurately than state of the art methods. Experiments also show that our model, starting from a set of binary labels, may discover rich information, such as different "schools of thought" amongst the annotators, and can group together images belonging to separate categories.

AB - Distributing labeling tasks among hundreds or thousands of annotators is an increasingly important method for annotating large datasets. We present a method for estimating the underlying value (e.g. the class) of each image from (noisy) annotations provided by multiple annotators. Our method is based on a model of the image formation and annotation process. Each image has different characteristics that are represented in an abstract Euclidean space. Each annotator is modeled as a multidimensional entity with variables representing competence, expertise and bias. This allows the model to discover and represent groups of annotators that have different sets of skills and knowledge, as well as groups of images that differ qualitatively. We find that our model predicts ground truth labels on both synthetic and real data more accurately than state of the art methods. Experiments also show that our model, starting from a set of binary labels, may discover rich information, such as different "schools of thought" amongst the annotators, and can group together images belonging to separate categories.

UR - http://www.scopus.com/inward/record.url?scp=84860645109&partnerID=8YFLogxK

M3 - Conference article

AN - SCOPUS:84860645109

JO - Advances in Neural Information Processing Systems 23: 24th Annual Conference on Neural Information Processing Systems 2010, NIPS 2010

JF - Advances in Neural Information Processing Systems 23: 24th Annual Conference on Neural Information Processing Systems 2010, NIPS 2010

T2 - 24th Annual Conference on Neural Information Processing Systems 2010, NIPS 2010

Y2 - 6 December 2010 through 9 December 2010

ER -

ID: 302047406