Is word order considered by foundation models? A comparative task-oriented analysis
Research output: Contribution to journal › Journal article › Research › peer-review
Word order, a linguistic concept essential for conveying accurate meaning, is seemingly not that necessary in language models based on the existing works. Contrary to this prevailing notion, our paper delves into the impacts of word order by employing carefully selected tasks that demand distinct abilities. Using three large language model families (ChatGPT, Claude, LLaMA), three controllable word order perturbation strategies, one novel perturbation qualification metric, four well-chosen tasks, and three languages, we conduct experiments to shed light on this topic. Empirical findings demonstrate that Foundation models take word order into consideration during generation. Moreover, tasks emphasizing reasoning abilities exhibit a greater reliance on word order compared to those primarily based on world knowledge.
Original language | English |
---|---|
Article number | 122700 |
Journal | Expert Systems with Applications |
Volume | 241 |
Number of pages | 10 |
ISSN | 0957-4174 |
DOIs | |
Publication status | Published - 2024 |
Bibliographical note
Publisher Copyright:
© 2023 Elsevier Ltd
- Foundation model, MGSM, Order perturbation ratio, WinoGrande, Word order
Research areas
ID: 378947148