Is word order considered by foundation models? A comparative task-oriented analysis
Publikation: Bidrag til tidsskrift › Tidsskriftartikel › Forskning › fagfællebedømt
Word order, a linguistic concept essential for conveying accurate meaning, is seemingly not that necessary in language models based on the existing works. Contrary to this prevailing notion, our paper delves into the impacts of word order by employing carefully selected tasks that demand distinct abilities. Using three large language model families (ChatGPT, Claude, LLaMA), three controllable word order perturbation strategies, one novel perturbation qualification metric, four well-chosen tasks, and three languages, we conduct experiments to shed light on this topic. Empirical findings demonstrate that Foundation models take word order into consideration during generation. Moreover, tasks emphasizing reasoning abilities exhibit a greater reliance on word order compared to those primarily based on world knowledge.
Originalsprog | Engelsk |
---|---|
Artikelnummer | 122700 |
Tidsskrift | Expert Systems with Applications |
Vol/bind | 241 |
Antal sider | 10 |
ISSN | 0957-4174 |
DOI | |
Status | Udgivet - 2024 |
Bibliografisk note
Funding Information:
We express our heartfelt gratitude to Anders Søgaard, Daniel Hershcovich, Laura Cabello Piqueras, and Yong Cao for their constructive discussions, insightful feedback, and valuable suggestions during this research. We also wish to extend our appreciation to the reviewers of this manuscript, whose insightful comments substantially improved the quality and clarity of this paper. We gratefully acknowledge the financial support from the China Scholarship Council (CSC) under Grant No. 202206020120 . Finally, we would like to acknowledge the support provided by the National Natural Science Foundation of China (NSFC) under Grant No. 61925203 .
Publisher Copyright:
© 2023 Elsevier Ltd
ID: 378947148