Generalized Quantifiers as a Source of Error in Multilingual NLU Benchmarks

Publikation: Bidrag til bog/antologi/rapport › Konferencebidrag i proceedings › Forskning › fagfællebedømt

Standard

Generalized Quantifiers as a Source of Error in Multilingual NLU Benchmarks. / Cui, Ruixiang; Hershcovich, Daniel; Søgaard, Anders.

NAACL 2022 - 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Proceedings of the Conference. Association for Computational Linguistics (ACL), 2022. s. 4875-4893.

Publikation: Bidrag til bog/antologi/rapport › Konferencebidrag i proceedings › Forskning › fagfællebedømt

Harvard

Cui, R, Hershcovich, D & Søgaard, A 2022, Generalized Quantifiers as a Source of Error in Multilingual NLU Benchmarks. i NAACL 2022 - 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Proceedings of the Conference. Association for Computational Linguistics (ACL), s. 4875-4893, 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2022, Seattle, USA, 10/07/2022. https://doi.org/10.18653/v1/2022.naacl-main.359

APA

Cui, R., Hershcovich, D., & Søgaard, A. (2022). Generalized Quantifiers as a Source of Error in Multilingual NLU Benchmarks. I NAACL 2022 - 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Proceedings of the Conference (s. 4875-4893). Association for Computational Linguistics (ACL). https://doi.org/10.18653/v1/2022.naacl-main.359

Vancouver

Cui R, Hershcovich D, Søgaard A. Generalized Quantifiers as a Source of Error in Multilingual NLU Benchmarks. I NAACL 2022 - 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Proceedings of the Conference. Association for Computational Linguistics (ACL). 2022. s. 4875-4893 https://doi.org/10.18653/v1/2022.naacl-main.359

Author

Cui, Ruixiang ; Hershcovich, Daniel ; Søgaard, Anders. / Generalized Quantifiers as a Source of Error in Multilingual NLU Benchmarks. NAACL 2022 - 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Proceedings of the Conference. Association for Computational Linguistics (ACL), 2022. s. 4875-4893

Bibtex

@inproceedings{b1d60801c9234e1b98f76f5da14eaf9b,

title = "Generalized Quantifiers as a Source of Error in Multilingual NLU Benchmarks",

abstract = "Logical approaches to representing language have developed and evaluated computational models of quantifier words since the 19th century, but today's NLU models still struggle to capture their semantics. We rely on Generalized Quantifier Theory for language-independent representations of the semantics of quantifier words, to quantify their contribution to the errors of NLU models. We find that quantifiers are pervasive in NLU benchmarks, and their occurrence at test time is associated with performance drops. Multilingual models also exhibit unsatisfying quantifier reasoning abilities, but not necessarily worse for non-English languages. To facilitate directly-targeted probing, we present an adversarial generalized quantifier NLI task (GQNLI) and show that pre-trained language models have a clear lack of robustness in generalized quantifier reasoning.",

author = "Ruixiang Cui and Daniel Hershcovich and Anders S{\o}gaard",

note = "Publisher Copyright: {\textcopyright} 2022 Association for Computational Linguistics.; 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2022 ; Conference date: 10-07-2022 Through 15-07-2022",

year = "2022",

doi = "10.18653/v1/2022.naacl-main.359",

language = "English",

pages = "4875--4893",

booktitle = "NAACL 2022 - 2022 Conference of the North American Chapter of the Association for Computational Linguistics",

publisher = "Association for Computational Linguistics (ACL)",

address = "United States",

}

RIS

TY - GEN

T1 - Generalized Quantifiers as a Source of Error in Multilingual NLU Benchmarks

AU - Cui, Ruixiang

AU - Hershcovich, Daniel

AU - Søgaard, Anders

PY - 2022

Y1 - 2022

N2 - Logical approaches to representing language have developed and evaluated computational models of quantifier words since the 19th century, but today's NLU models still struggle to capture their semantics. We rely on Generalized Quantifier Theory for language-independent representations of the semantics of quantifier words, to quantify their contribution to the errors of NLU models. We find that quantifiers are pervasive in NLU benchmarks, and their occurrence at test time is associated with performance drops. Multilingual models also exhibit unsatisfying quantifier reasoning abilities, but not necessarily worse for non-English languages. To facilitate directly-targeted probing, we present an adversarial generalized quantifier NLI task (GQNLI) and show that pre-trained language models have a clear lack of robustness in generalized quantifier reasoning.

AB - Logical approaches to representing language have developed and evaluated computational models of quantifier words since the 19th century, but today's NLU models still struggle to capture their semantics. We rely on Generalized Quantifier Theory for language-independent representations of the semantics of quantifier words, to quantify their contribution to the errors of NLU models. We find that quantifiers are pervasive in NLU benchmarks, and their occurrence at test time is associated with performance drops. Multilingual models also exhibit unsatisfying quantifier reasoning abilities, but not necessarily worse for non-English languages. To facilitate directly-targeted probing, we present an adversarial generalized quantifier NLI task (GQNLI) and show that pre-trained language models have a clear lack of robustness in generalized quantifier reasoning.

UR - http://www.scopus.com/inward/record.url?scp=85138393396&partnerID=8YFLogxK

U2 - 10.18653/v1/2022.naacl-main.359

DO - 10.18653/v1/2022.naacl-main.359

M3 - Article in proceedings

AN - SCOPUS:85138393396

SP - 4875

EP - 4893

BT - NAACL 2022 - 2022 Conference of the North American Chapter of the Association for Computational Linguistics

PB - Association for Computational Linguistics (ACL)

T2 - 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2022

Y2 - 10 July 2022 through 15 July 2022

ER -

ID: 339850247

Datalogisk Institut