TY - JOUR

T1 - What makes the difference?

T2 - An empirical comparison of fusion strategies for multimodal language analysis

AU - Gkoumas, Dimitris

AU - Li, Qiuchi

AU - Lioma, Christina

AU - Yu, Yijun

AU - Song, Dawei

N1 - Publisher Copyright: © 2020 Elsevier B.V.

PY - 2021

Y1 - 2021

N2 - Multimodal video sentiment analysis is a rapidly growing area. It combines verbal (i.e., linguistic) and non-verbal modalities (i.e., visual, acoustic) to predict the sentiment of utterances. A recent trend has been geared towards different modality fusion models utilizing various attention, memory and recurrent components. However, there lacks a systematic investigation on how these different components contribute to solving the problem as well as their limitations. This paper aims to fill the gap, marking the following key innovations. We present the first large-scale and comprehensive empirical comparison of eleven state-of-the-art (SOTA) modality fusion approaches in two video sentiment analysis tasks, with three SOTA benchmark corpora. An in-depth analysis of the results shows that the attention mechanisms are the most effective for modelling crossmodal interactions, yet they are computationally expensive. Second, additional levels of crossmodal interaction decrease performance. Third, positive sentiment utterances are the most challenging cases for all approaches. Finally, integrating context and utilizing the linguistic modality as a pivot for non-verbal modalities improve performance. We expect that the findings would provide helpful insights and guidance to the development of more effective modality fusion models.

AB - Multimodal video sentiment analysis is a rapidly growing area. It combines verbal (i.e., linguistic) and non-verbal modalities (i.e., visual, acoustic) to predict the sentiment of utterances. A recent trend has been geared towards different modality fusion models utilizing various attention, memory and recurrent components. However, there lacks a systematic investigation on how these different components contribute to solving the problem as well as their limitations. This paper aims to fill the gap, marking the following key innovations. We present the first large-scale and comprehensive empirical comparison of eleven state-of-the-art (SOTA) modality fusion approaches in two video sentiment analysis tasks, with three SOTA benchmark corpora. An in-depth analysis of the results shows that the attention mechanisms are the most effective for modelling crossmodal interactions, yet they are computationally expensive. Second, additional levels of crossmodal interaction decrease performance. Third, positive sentiment utterances are the most challenging cases for all approaches. Finally, integrating context and utilizing the linguistic modality as a pivot for non-verbal modalities improve performance. We expect that the findings would provide helpful insights and guidance to the development of more effective modality fusion models.

KW - Emotion recognition

KW - Multimodal human language understanding

KW - Reproducibility in multimodal machine learning

KW - Video sentiment analysis

UR - http://www.scopus.com/inward/record.url?scp=85091217348&partnerID=8YFLogxK

U2 - 10.1016/j.inffus.2020.09.005

DO - 10.1016/j.inffus.2020.09.005

M3 - Journal article

AN - SCOPUS:85091217348

VL - 66

SP - 184

EP - 197

JO - Information Fusion

JF - Information Fusion

SN - 1566-2535

ER -