A Multilingual Benchmark for Probing Negation-Awareness with Minimal Pairs

Publikation: Bidrag til bog/antologi/rapport › Konferencebidrag i proceedings › Forskning › fagfællebedømt

Standard

A Multilingual Benchmark for Probing Negation-Awareness with Minimal Pairs. / Hartmann, Mareike; de Lhoneux, Miryam; Hershcovich, Daniel; Kementchedjhieva, Yova Radoslavova; Nielsen, Lukas Christian; Qiu, Chen; Søgaard, Anders.

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 2021. s. 244–257.

Publikation: Bidrag til bog/antologi/rapport › Konferencebidrag i proceedings › Forskning › fagfællebedømt

Harvard

Hartmann, M, de Lhoneux, M, Hershcovich, D, Kementchedjhieva, YR, Nielsen, LC, Qiu, C & Søgaard, A 2021, A Multilingual Benchmark for Probing Negation-Awareness with Minimal Pairs. i Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, s. 244–257, 2021 Conference on Empirical Methods in Natural Language Processing, 07/11/2021. https://doi.org/10.18653/v1/2021.conll-1.19

APA

Hartmann, M., de Lhoneux, M., Hershcovich, D., Kementchedjhieva, Y. R., Nielsen, L. C., Qiu, C., & Søgaard, A. (2021). A Multilingual Benchmark for Probing Negation-Awareness with Minimal Pairs. I Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (s. 244–257). Association for Computational Linguistics. https://doi.org/10.18653/v1/2021.conll-1.19

Vancouver

Hartmann M, de Lhoneux M, Hershcovich D, Kementchedjhieva YR, Nielsen LC, Qiu C o.a. A Multilingual Benchmark for Probing Negation-Awareness with Minimal Pairs. I Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics. 2021. s. 244–257 https://doi.org/10.18653/v1/2021.conll-1.19

Author

Hartmann, Mareike ; de Lhoneux, Miryam ; Hershcovich, Daniel ; Kementchedjhieva, Yova Radoslavova ; Nielsen, Lukas Christian ; Qiu, Chen ; Søgaard, Anders. / A Multilingual Benchmark for Probing Negation-Awareness with Minimal Pairs. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 2021. s. 244–257

Bibtex

@inproceedings{3768eada16434875b3486afd8b75919c,
title = "A Multilingual Benchmark for Probing Negation-Awareness with Minimal Pairs",
abstract = "Negation is one of the most fundamental concepts in human cognition and language, and several natural language inference (NLI) probes have been designed to investigate pretrained language models{\textquoteright} ability to detect and reason with negation. However, the existing probing datasets are limited to English only, and do not enable controlled probing of performance in the absence or presence of negation. In response, we present a multilingual (English, Bulgarian, German, French and Chinese) benchmark collection of NLI examples that are grammatical and correctly labeled, as a result of manual inspection and reformulation. We use the benchmark to probe the negation-awareness of multilingual language models and find that models that correctly predict examples with negation cues, often fail to correctly predict their counter-examples without negation cues, even when the cues are irrelevant for semantic inference.",
author = "Mareike Hartmann and {de Lhoneux}, Miryam and Daniel Hershcovich and Kementchedjhieva, {Yova Radoslavova} and Nielsen, {Lukas Christian} and Chen Qiu and Anders S{\o}gaard",
year = "2021",
doi = "10.18653/v1/2021.conll-1.19",
language = "English",
pages = "244–257",
booktitle = "Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing",
publisher = "Association for Computational Linguistics",
note = "2021 Conference on Empirical Methods in Natural Language Processing ; Conference date: 07-11-2021 Through 11-11-2021",

}

RIS

TY - GEN

T1 - A Multilingual Benchmark for Probing Negation-Awareness with Minimal Pairs

AU - Hartmann, Mareike

AU - de Lhoneux, Miryam

AU - Hershcovich, Daniel

AU - Kementchedjhieva, Yova Radoslavova

AU - Nielsen, Lukas Christian

AU - Qiu, Chen

AU - Søgaard, Anders

PY - 2021

Y1 - 2021

N2 - Negation is one of the most fundamental concepts in human cognition and language, and several natural language inference (NLI) probes have been designed to investigate pretrained language models’ ability to detect and reason with negation. However, the existing probing datasets are limited to English only, and do not enable controlled probing of performance in the absence or presence of negation. In response, we present a multilingual (English, Bulgarian, German, French and Chinese) benchmark collection of NLI examples that are grammatical and correctly labeled, as a result of manual inspection and reformulation. We use the benchmark to probe the negation-awareness of multilingual language models and find that models that correctly predict examples with negation cues, often fail to correctly predict their counter-examples without negation cues, even when the cues are irrelevant for semantic inference.

AB - Negation is one of the most fundamental concepts in human cognition and language, and several natural language inference (NLI) probes have been designed to investigate pretrained language models’ ability to detect and reason with negation. However, the existing probing datasets are limited to English only, and do not enable controlled probing of performance in the absence or presence of negation. In response, we present a multilingual (English, Bulgarian, German, French and Chinese) benchmark collection of NLI examples that are grammatical and correctly labeled, as a result of manual inspection and reformulation. We use the benchmark to probe the negation-awareness of multilingual language models and find that models that correctly predict examples with negation cues, often fail to correctly predict their counter-examples without negation cues, even when the cues are irrelevant for semantic inference.

U2 - 10.18653/v1/2021.conll-1.19

DO - 10.18653/v1/2021.conll-1.19

M3 - Article in proceedings

SP - 244

EP - 257

BT - Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing

PB - Association for Computational Linguistics

T2 - 2021 Conference on Empirical Methods in Natural Language Processing

Y2 - 7 November 2021 through 11 November 2021

ER -

ID: 299825199