Generalized Quantifiers as a Source of Error in Multilingual NLU Benchmarks
Research output: Chapter in Book/Report/Conference proceeding › Article in proceedings › Research › peer-review
Documents
- Fulltext
Final published version, 434 KB, PDF document
Logical approaches to representing language have developed and evaluated computational models of quantifier words since the 19th century, but today's NLU models still struggle to capture their semantics. We rely on Generalized Quantifier Theory for language-independent representations of the semantics of quantifier words, to quantify their contribution to the errors of NLU models. We find that quantifiers are pervasive in NLU benchmarks, and their occurrence at test time is associated with performance drops. Multilingual models also exhibit unsatisfying quantifier reasoning abilities, but not necessarily worse for non-English languages. To facilitate directly-targeted probing, we present an adversarial generalized quantifier NLI task (GQNLI) and show that pre-trained language models have a clear lack of robustness in generalized quantifier reasoning.
Original language | English |
---|---|
Title of host publication | NAACL 2022 - 2022 Conference of the North American Chapter of the Association for Computational Linguistics : Human Language Technologies, Proceedings of the Conference |
Publisher | Association for Computational Linguistics (ACL) |
Publication date | 2022 |
Pages | 4875-4893 |
ISBN (Electronic) | 9781955917711 |
DOIs | |
Publication status | Published - 2022 |
Event | 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2022 - Seattle, United States Duration: 10 Jul 2022 → 15 Jul 2022 |
Conference
Conference | 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2022 |
---|---|
Land | United States |
By | Seattle |
Periode | 10/07/2022 → 15/07/2022 |
Sponsor | Amazon, Bloomberg, et al., Google Research, LIVE PERSON, Meta |
Bibliographical note
Publisher Copyright:
© 2022 Association for Computational Linguistics.
ID: 339850247