Multimodal intelligence: Representation learning, information fusion, and applications

C Zhang, Z Yang, X He, L Deng - IEEE Journal of Selected …, 2020‏ - ieeexplore.ieee.org
Deep learning methods haverevolutionized speech recognition, image recognition, and
natural language processing since 2010. Each of these tasks involves a single modality in …

Multimodal machine learning: A survey and taxonomy

T Baltrušaitis, C Ahuja… - IEEE transactions on …, 2018‏ - ieeexplore.ieee.org
Our experience of the world is multimodal-we see objects, hear sounds, feel texture, smell
odors, and taste flavors. Modality refers to the way in which something happens or is …

Multimodal sentiment analysis based on fusion methods: A survey

L Zhu, Z Zhu, C Zhang, Y Xu, X Kong - Information Fusion, 2023‏ - Elsevier
Sentiment analysis is an emerging technology that aims to explore people's attitudes toward
an entity. It can be applied in a variety of different fields and scenarios, such as product …

Expansion-squeeze-excitation fusion network for elderly activity recognition

X Shu, J Yang, R Yan, Y Song - IEEE Transactions on Circuits …, 2022‏ - ieeexplore.ieee.org
This work focuses on the task of elderly activity recognition, which is a challenging task due
to the existence of individual actions and human-object interactions in elderly activities …

Multimodal language analysis in the wild: Cmu-mosei dataset and interpretable dynamic fusion graph

AAB Zadeh, PP Liang, S Poria, E Cambria… - Proceedings of the …, 2018‏ - aclanthology.org
Analyzing human multimodal language is an emerging area of research in NLP. Intrinsically
this language is multimodal (heterogeneous), sequential and asynchronous; it consists of …

Hybrid contrastive learning of tri-modal representation for multimodal sentiment analysis

S Mai, Y Zeng, S Zheng, H Hu - IEEE Transactions on Affective …, 2022‏ - ieeexplore.ieee.org
The wide application of smart devices enables the availability of multimodal data, which can
be utilized in many tasks. In the field of multimodal sentiment analysis, most previous works …

Efficient low-rank multimodal fusion with modality-specific factors

Z Liu, Y Shen, VB Lakshminarasimhan… - arxiv preprint arxiv …, 2018‏ - arxiv.org
Multimodal research is an emerging field of artificial intelligence, and one of the main
research problems in this field is multimodal fusion. The fusion of multimodal data is the …

Memory fusion network for multi-view sequential learning

A Zadeh, PP Liang, N Mazumder, S Poria… - Proceedings of the …, 2018‏ - ojs.aaai.org
Multi-view sequential learning is a fundamental problem in machine learning dealing with
multi-view sequences. In a multi-view sequence, there exists two forms of interactions …

Words can shift: Dynamically adjusting word representations using nonverbal behaviors

Y Wang, Y Shen, Z Liu, PP Liang, A Zadeh… - Proceedings of the …, 2019‏ - ojs.aaai.org
Humans convey their intentions through the usage of both verbal and nonverbal behaviors
during face-to-face communication. Speaker intentions often vary dynamically depending on …

Found in translation: Learning robust joint representations by cyclic translations between modalities

H Pham, PP Liang, T Manzini, LP Morency… - Proceedings of the …, 2019‏ - ojs.aaai.org
Multimodal sentiment analysis is a core research area that studies speaker sentiment
expressed from the language, visual, and acoustic modalities. The central challenge in …