Common Sense Bias in Semantic Role Labeling
Publikation: Bidrag til bog/antologi/rapport › Konferencebidrag i proceedings › Forskning › fagfællebedømt
Dokumenter
- Fulltext
Forlagets udgivne version, 295 KB, PDF-dokument
Large-scale language models such as ELMo and BERT have pushed the horizon of what is possible in semantic role labeling (SRL), solving the out-of-vocabulary problem and enabling end-to-end systems, but they have also introduced significant biases. We evaluate three SRL parsers on very simple transitive sentences with verbs usually associated with animate subjects and objects, such as, “Mary babysat Tom”: a state-of-the-art parser based on BERT, an older parser based on GloVe, and an even older parser from before the days of word embeddings. When arguments are word forms predominantly used as person names, aligning with common sense expectations of animacy, the BERT-based parser is unsurprisingly superior; yet, with abstract or random nouns, the opposite picture emerges. We refer to this as “common sense bias” and present a challenge dataset for evaluating the extent to which parsers are sensitive to such a bias. Our code and challenge dataset are available here: github.com/coastalcph/comte
Originalsprog | Engelsk |
---|---|
Titel | Proceedings of the Seventh Workshop on Noisy User-generated Text (W-NUT 2021) |
Forlag | Association for Computational Linguistics |
Publikationsdato | 2021 |
Sider | 114–119 |
DOI | |
Status | Udgivet - 2021 |
Begivenhed | 7th Workshop on Noisy User-generated Text (W-NUT 2021) - Online Varighed: 11 nov. 2021 → 11 nov. 2021 |
Konference
Konference | 7th Workshop on Noisy User-generated Text (W-NUT 2021) |
---|---|
By | Online |
Periode | 11/11/2021 → 11/11/2021 |
ID: 300076700