Machine Reading, Fast and Slow: When Do Models “Understand” Language?
Research output: Chapter in Book/Report/Conference proceeding › Article in proceedings › Research › peer-review
Documents
- Fulltext
Final published version, 461 KB, PDF document
Two of the most fundamental issues in Natural Language Understanding (NLU) at present are: (a) how it can established whether deep learning-based models score highly on NLU benchmarks for the ”right” reasons; and (b) what those reasons would even be. We investigate the behavior of reading comprehension models with respect to two linguistic ”skills”: coreference resolution and comparison. We propose a definition for the reasoning steps expected from a system that would be ”reading slowly”, and compare that with the behavior of five models of the BERT family of various sizes, observed through saliency scores and counterfactual explanations. We find that for comparison (but not coreference) the systems based on larger encoders are more likely to rely on the ”right” information, but even they struggle with generalization, suggesting that they still learn specific lexical patterns rather than the general principles of comparison.
Original language | English |
---|---|
Title of host publication | Proceedings of the 29th International Conference on Computational Linguistics |
Publisher | Association for Computational Linguistics (ACL) |
Publication date | 2022 |
Pages | 78–93 |
Publication status | Published - 2022 |
Event | THE 29TH INTERNATIONAL CONFERENCE ON COMPUTATIONAL LINGUISTICS - Gyeongju, Republic of Korea Duration: 12 Oct 2022 → 17 Oct 2022 |
Conference
Conference | THE 29TH INTERNATIONAL CONFERENCE ON COMPUTATIONAL LINGUISTICS |
---|---|
Location | Gyeongju, Republic of Korea |
Periode | 12/10/2022 → 17/10/2022 |
Links
- https://aclanthology.org/2022.coling-1.8/
Final published version
ID: 341057090