Quantum-inspired multimodal fusion for video sentiment analysis

Publikation: Bidrag til tidsskrift › Tidsskriftartikel › Forskning › fagfællebedømt

Standard

Quantum-inspired multimodal fusion for video sentiment analysis. / Li, Qiuchi; Gkoumas, Dimitris; Lioma, Christina; Melucci, Massimo.

I: Information Fusion, Bind 65, 2021, s. 58-71.

Publikation: Bidrag til tidsskrift › Tidsskriftartikel › Forskning › fagfællebedømt

Harvard

Li, Q, Gkoumas, D, Lioma, C & Melucci, M 2021, 'Quantum-inspired multimodal fusion for video sentiment analysis', Information Fusion, bind 65, s. 58-71. https://doi.org/10.1016/j.inffus.2020.08.006

APA

Li, Q., Gkoumas, D., Lioma, C., & Melucci, M. (2021). Quantum-inspired multimodal fusion for video sentiment analysis. Information Fusion, 65, 58-71. https://doi.org/10.1016/j.inffus.2020.08.006

Vancouver

Li Q, Gkoumas D, Lioma C, Melucci M. Quantum-inspired multimodal fusion for video sentiment analysis. Information Fusion. 2021;65:58-71. https://doi.org/10.1016/j.inffus.2020.08.006

Author

Li, Qiuchi ; Gkoumas, Dimitris ; Lioma, Christina ; Melucci, Massimo. / Quantum-inspired multimodal fusion for video sentiment analysis. I: Information Fusion. 2021 ; Bind 65. s. 58-71.

Bibtex

@article{ba44a1cca1034eeaac541880e4ef80a9,

title = "Quantum-inspired multimodal fusion for video sentiment analysis",

abstract = "We tackle the crucial challenge of fusing different modalities of features for multimodal sentiment analysis. Mainly based on neural networks, existing approaches largely model multimodal interactions in an implicit and hard-to-understand manner. We address this limitation with inspirations from quantum theory, which contains principled methods for modeling complicated interactions and correlations. In our quantum-inspired framework, the word interaction within a single modality and the interaction across modalities are formulated with superposition and entanglement respectively at different stages. The complex-valued neural network implementation of the framework achieves comparable results to state-of-the-art systems on two benchmarking video sentiment analysis datasets. In the meantime, we produce the unimodal and bimodal sentiment directly from the model to interpret the entangled decision.",

keywords = "Machine learning, Multimodal sentiment analysis, Quantum theory",

author = "Qiuchi Li and Dimitris Gkoumas and Christina Lioma and Massimo Melucci",

note = "Publisher Copyright: {\textcopyright} 2020 Elsevier B.V.",

year = "2021",

doi = "10.1016/j.inffus.2020.08.006",

language = "English",

volume = "65",

pages = "58--71",

journal = "Information Fusion",

issn = "1566-2535",

publisher = "Elsevier",

}

RIS

TY - JOUR

T1 - Quantum-inspired multimodal fusion for video sentiment analysis

AU - Li, Qiuchi

AU - Gkoumas, Dimitris

AU - Lioma, Christina

AU - Melucci, Massimo

PY - 2021

Y1 - 2021

N2 - We tackle the crucial challenge of fusing different modalities of features for multimodal sentiment analysis. Mainly based on neural networks, existing approaches largely model multimodal interactions in an implicit and hard-to-understand manner. We address this limitation with inspirations from quantum theory, which contains principled methods for modeling complicated interactions and correlations. In our quantum-inspired framework, the word interaction within a single modality and the interaction across modalities are formulated with superposition and entanglement respectively at different stages. The complex-valued neural network implementation of the framework achieves comparable results to state-of-the-art systems on two benchmarking video sentiment analysis datasets. In the meantime, we produce the unimodal and bimodal sentiment directly from the model to interpret the entangled decision.

AB - We tackle the crucial challenge of fusing different modalities of features for multimodal sentiment analysis. Mainly based on neural networks, existing approaches largely model multimodal interactions in an implicit and hard-to-understand manner. We address this limitation with inspirations from quantum theory, which contains principled methods for modeling complicated interactions and correlations. In our quantum-inspired framework, the word interaction within a single modality and the interaction across modalities are formulated with superposition and entanglement respectively at different stages. The complex-valued neural network implementation of the framework achieves comparable results to state-of-the-art systems on two benchmarking video sentiment analysis datasets. In the meantime, we produce the unimodal and bimodal sentiment directly from the model to interpret the entangled decision.

KW - Machine learning

KW - Multimodal sentiment analysis

KW - Quantum theory

U2 - 10.1016/j.inffus.2020.08.006

DO - 10.1016/j.inffus.2020.08.006

M3 - Journal article

AN - SCOPUS:85089582013

VL - 65

SP - 58

EP - 71

JO - Information Fusion

JF - Information Fusion

SN - 1566-2535

ER -

ID: 306691917

Datalogisk Institut