Multi-head Self-attention with Role-Guided Masks
Publikation: Bidrag til bog/antologi/rapport › Konferencebidrag i proceedings › Forskning › fagfællebedømt
Dokumenter
- Fulltext
Indsendt manuskript, 218 KB, PDF-dokument
The state of the art in learning meaningful semantic representations of words is the Transformer model and its attention mechanisms. Simply put, the attention mechanisms learn to attend to specific parts of the input dispensing recurrence and convolutions. While some of the learned attention heads have been found to play linguistically interpretable roles, they can be redundant or prone to errors. We propose a method to guide the attention heads towards roles identified in prior work as important. We do this by defining role-specific masks to constrain the heads to attend to specific parts of the input, such that different heads are designed to play different roles. Experiments on text classification and machine translation using 7 different datasets show that our method outperforms competitive attention-based, CNN, and RNN baselines.
Originalsprog | Engelsk |
---|---|
Titel | Advances in Information Retrieval - 43rd European Conference on IR Research, ECIR 2021, Proceedings, Part II |
Redaktører | Djoerd Hiemstra, Marie-Francine Moens, Josiane Mothe, Raffaele Perego, Martin Potthast, Fabrizio Sebastiani |
Forlag | Springer |
Publikationsdato | 2021 |
Sider | 432-439 |
ISBN (Trykt) | 9783030722395 |
DOI | |
Status | Udgivet - 2021 |
Begivenhed | 43rd European Conference on Information Retrieval, ECIR 2021 - Virtual, Online Varighed: 28 mar. 2021 → 1 apr. 2021 |
Konference
Konference | 43rd European Conference on Information Retrieval, ECIR 2021 |
---|---|
By | Virtual, Online |
Periode | 28/03/2021 → 01/04/2021 |
Navn | Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) |
---|---|
Vol/bind | 12657 LNCS |
ISSN | 0302-9743 |
Bibliografisk note
Funding Information:
Acknowledgments. This work is supported by the European Union’s Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie grant agreement No. 721321 (QUARTZ project) and No. 893667 (METER project).
Publisher Copyright:
© 2021, Springer Nature Switzerland AG.
ID: 283133892