Word Order's Impacts: Insights from Reordering and Generation Analysis

Research output: Working paper › Preprint › Research

Standard

Word Order's Impacts : Insights from Reordering and Generation Analysis. / Zhao, Qinghua; Li, Jiaang; Li, Lei; Zhou, Zenghui; Liu, Junfeng.

arXiv.org, 2024.

Research output: Working paper › Preprint › Research

Harvard

Zhao, Q, Li, J, Li, L, Zhou, Z & Liu, J 2024 'Word Order's Impacts: Insights from Reordering and Generation Analysis' arXiv.org. <https://arxiv.org/abs/2403.11473>

APA

Zhao, Q., Li, J., Li, L., Zhou, Z., & Liu, J. (2024). Word Order's Impacts: Insights from Reordering and Generation Analysis. arXiv.org. https://arxiv.org/abs/2403.11473

Vancouver

Zhao Q, Li J, Li L, Zhou Z, Liu J. Word Order's Impacts: Insights from Reordering and Generation Analysis. arXiv.org. 2024 Mar 18.

Author

Zhao, Qinghua ; Li, Jiaang ; Li, Lei ; Zhou, Zenghui ; Liu, Junfeng. / Word Order's Impacts : Insights from Reordering and Generation Analysis. arXiv.org, 2024.

Bibtex

@techreport{e965caf296904dd89d79961b41723791,
title = "Word Order's Impacts: Insights from Reordering and Generation Analysis",
abstract = " Existing works have studied the impacts of the order of words within natural text. They usually analyze it by destroying the original order of words to create a scrambled sequence, and then comparing the models' performance between the original and scrambled sequences. The experimental results demonstrate marginal drops. Considering this findings, different hypothesis about word order is proposed, including ``the order of words is redundant with lexical semantics'', and ``models do not rely on word order''. In this paper, we revisit the aforementioned hypotheses by adding a order reconstruction perspective, and selecting datasets of different spectrum. Specifically, we first select four different datasets, and then design order reconstruction and continuing generation tasks. Empirical findings support that ChatGPT relies on word order to infer, but cannot support or negate the redundancy relations between word order lexical semantics. ",
keywords = "cs.CL, cs.AI",
author = "Qinghua Zhao and Jiaang Li and Lei Li and Zenghui Zhou and Junfeng Liu",
year = "2024",
month = mar,
day = "18",
language = "Udefineret/Ukendt",
publisher = "arXiv.org",
type = "WorkingPaper",
institution = "arXiv.org",

}

RIS

TY - UNPB

T1 - Word Order's Impacts

T2 - Insights from Reordering and Generation Analysis

AU - Zhao, Qinghua

AU - Li, Jiaang

AU - Li, Lei

AU - Zhou, Zenghui

AU - Liu, Junfeng

PY - 2024/3/18

Y1 - 2024/3/18

N2 - Existing works have studied the impacts of the order of words within natural text. They usually analyze it by destroying the original order of words to create a scrambled sequence, and then comparing the models' performance between the original and scrambled sequences. The experimental results demonstrate marginal drops. Considering this findings, different hypothesis about word order is proposed, including ``the order of words is redundant with lexical semantics'', and ``models do not rely on word order''. In this paper, we revisit the aforementioned hypotheses by adding a order reconstruction perspective, and selecting datasets of different spectrum. Specifically, we first select four different datasets, and then design order reconstruction and continuing generation tasks. Empirical findings support that ChatGPT relies on word order to infer, but cannot support or negate the redundancy relations between word order lexical semantics.

AB - Existing works have studied the impacts of the order of words within natural text. They usually analyze it by destroying the original order of words to create a scrambled sequence, and then comparing the models' performance between the original and scrambled sequences. The experimental results demonstrate marginal drops. Considering this findings, different hypothesis about word order is proposed, including ``the order of words is redundant with lexical semantics'', and ``models do not rely on word order''. In this paper, we revisit the aforementioned hypotheses by adding a order reconstruction perspective, and selecting datasets of different spectrum. Specifically, we first select four different datasets, and then design order reconstruction and continuing generation tasks. Empirical findings support that ChatGPT relies on word order to infer, but cannot support or negate the redundancy relations between word order lexical semantics.

KW - cs.CL

KW - cs.AI

M3 - Preprint

BT - Word Order's Impacts

PB - arXiv.org

ER -

ID: 395360718