Evaluation of Summarization Systems across Gender, Age, and Race

Publikation: Bidrag til bog/antologi/rapportKonferencebidrag i proceedingsForskningfagfællebedømt

Dokumenter

Summarization systems are ultimately evaluated by human annotators and raters. Usually, annotators and raters do not reflect the demographics of end users, but are recruited through student populations or crowdsourcing platforms with skewed demographics. For two different evaluation scenarios – evaluation against gold summaries and system output ratings – we show that summary evaluation is sensitive to protected attributes. This can severely bias system development and evaluation, leading us to build models that cater for some groups rather than others.
OriginalsprogEngelsk
TitelProceedings of the Third Workshop on New Frontiers in Summarization
ForlagAssociation for Computational Linguistics
Publikationsdato2021
Sider51–56
DOI
StatusUdgivet - 2021
Begivenhed3rd Workshop on New Frontiers in Summarization - Online
Varighed: 10 nov. 202110 nov. 2021

Konference

Konference3rd Workshop on New Frontiers in Summarization
ByOnline
Periode10/11/202110/11/2021

ID: 300074299