Is word order considered by foundation models? A comparative task-oriented analysis

Publikation: Bidrag til tidsskrift › Tidsskriftartikel › Forskning › fagfællebedømt

Standard

Is word order considered by foundation models? A comparative task-oriented analysis. / Zhao, Qinghua; Li, Jiaang; Liu, Junfeng; Kang, Zhongfeng; Zhou, Zenghui.

I: Expert Systems with Applications, Bind 241, 122700, 2024.

Publikation: Bidrag til tidsskrift › Tidsskriftartikel › Forskning › fagfællebedømt

Harvard

Zhao, Q, Li, J, Liu, J, Kang, Z & Zhou, Z 2024, 'Is word order considered by foundation models? A comparative task-oriented analysis', Expert Systems with Applications, bind 241, 122700. https://doi.org/10.1016/j.eswa.2023.122700

APA

Zhao, Q., Li, J., Liu, J., Kang, Z., & Zhou, Z. (2024). Is word order considered by foundation models? A comparative task-oriented analysis. Expert Systems with Applications, 241, [122700]. https://doi.org/10.1016/j.eswa.2023.122700

Vancouver

Zhao Q, Li J, Liu J, Kang Z, Zhou Z. Is word order considered by foundation models? A comparative task-oriented analysis. Expert Systems with Applications. 2024;241. 122700. https://doi.org/10.1016/j.eswa.2023.122700

Author

Zhao, Qinghua ; Li, Jiaang ; Liu, Junfeng ; Kang, Zhongfeng ; Zhou, Zenghui. / Is word order considered by foundation models? A comparative task-oriented analysis. I: Expert Systems with Applications. 2024 ; Bind 241.

Bibtex

@article{a58a617f6ce1420f80b4f7a501e2d9e6,

title = "Is word order considered by foundation models? A comparative task-oriented analysis",

abstract = "Word order, a linguistic concept essential for conveying accurate meaning, is seemingly not that necessary in language models based on the existing works. Contrary to this prevailing notion, our paper delves into the impacts of word order by employing carefully selected tasks that demand distinct abilities. Using three large language model families (ChatGPT, Claude, LLaMA), three controllable word order perturbation strategies, one novel perturbation qualification metric, four well-chosen tasks, and three languages, we conduct experiments to shed light on this topic. Empirical findings demonstrate that Foundation models take word order into consideration during generation. Moreover, tasks emphasizing reasoning abilities exhibit a greater reliance on word order compared to those primarily based on world knowledge.",

keywords = "Foundation model, MGSM, Order perturbation ratio, WinoGrande, Word order",

author = "Qinghua Zhao and Jiaang Li and Junfeng Liu and Zhongfeng Kang and Zenghui Zhou",

note = "Publisher Copyright: {\textcopyright} 2023 Elsevier Ltd",

year = "2024",

doi = "10.1016/j.eswa.2023.122700",

language = "English",

volume = "241",

journal = "Expert Systems with Applications",

issn = "0957-4174",

publisher = "Pergamon Press",

}

RIS

TY - JOUR

T1 - Is word order considered by foundation models? A comparative task-oriented analysis

AU - Zhao, Qinghua

AU - Li, Jiaang

AU - Liu, Junfeng

AU - Kang, Zhongfeng

AU - Zhou, Zenghui

PY - 2024

Y1 - 2024

N2 - Word order, a linguistic concept essential for conveying accurate meaning, is seemingly not that necessary in language models based on the existing works. Contrary to this prevailing notion, our paper delves into the impacts of word order by employing carefully selected tasks that demand distinct abilities. Using three large language model families (ChatGPT, Claude, LLaMA), three controllable word order perturbation strategies, one novel perturbation qualification metric, four well-chosen tasks, and three languages, we conduct experiments to shed light on this topic. Empirical findings demonstrate that Foundation models take word order into consideration during generation. Moreover, tasks emphasizing reasoning abilities exhibit a greater reliance on word order compared to those primarily based on world knowledge.

AB - Word order, a linguistic concept essential for conveying accurate meaning, is seemingly not that necessary in language models based on the existing works. Contrary to this prevailing notion, our paper delves into the impacts of word order by employing carefully selected tasks that demand distinct abilities. Using three large language model families (ChatGPT, Claude, LLaMA), three controllable word order perturbation strategies, one novel perturbation qualification metric, four well-chosen tasks, and three languages, we conduct experiments to shed light on this topic. Empirical findings demonstrate that Foundation models take word order into consideration during generation. Moreover, tasks emphasizing reasoning abilities exhibit a greater reliance on word order compared to those primarily based on world knowledge.

KW - Foundation model

KW - MGSM

KW - Order perturbation ratio

KW - WinoGrande

KW - Word order

U2 - 10.1016/j.eswa.2023.122700

DO - 10.1016/j.eswa.2023.122700

M3 - Journal article

AN - SCOPUS:85178663660

VL - 241

JO - Expert Systems with Applications

JF - Expert Systems with Applications

SN - 0957-4174

M1 - 122700

ER -

ID: 378947148

Datalogisk Institut