Partial information decomposition for continuous variables based on shared exclusions: Analytical formulation and estimation

DA Ehrlich, K Schick-Poland, A Makkeh, F Lanfermann… - Physical Review E, 2024 - APS
Describing statistical dependencies is foundational to empirical scientific research. For
uncovering intricate and possibly nonlinear dependencies between a single target variable …

Multimodal fusion interactions: A study of human and automatic quantification

PP Liang, Y Cheng, R Salakhutdinov… - Proceedings of the 25th …, 2023 - dl.acm.org
In order to perform multimodal fusion of heterogeneous signals, we need to understand their
interactions: how each modality individually provides information useful for a task and how …

Quantifying spuriousness of biased datasets using partial information decomposition

B Halder, F Hamman, P Dissanayake, Q Zhang… - arxiv preprint arxiv …, 2024 - arxiv.org
Spurious patterns refer to a mathematical association between two or more variables in a
dataset that are not causally related. However, this notion of spuriousness, which is usually …

What should a neuron aim for? Designing local objective functions based on information theory

AC Schneider, V Neuhaus, DA Ehrlich… - arxiv preprint arxiv …, 2024 - arxiv.org
In modern deep neural networks, the learning dynamics of the individual neurons is often
obscure, as the networks are trained via global optimization. Conversely, biological systems …

MMoE: Enhancing Multimodal Models with Mixtures of Multimodal Interaction Experts

H Yu, Z Qi, L Jang, R Salakhutdinov… - Proceedings of the …, 2024 - aclanthology.org
Advances in multimodal models have greatly improved how interactions relevant to various
tasks are modeled. Today's multimodal models mainly focus on the correspondence …

Towards Modality Generalization: A Benchmark and Prospective Analysis

X Liu, X **a, Z Huang, TS Chua - arxiv preprint arxiv:2412.18277, 2024 - arxiv.org
Multi-modal learning has achieved remarkable success by integrating information from
various modalities, achieving superior performance in tasks like recognition and retrieval …

What to align in multimodal contrastive learning?

B Dufumier, J Castillo-Navarro, D Tuia… - arxiv preprint arxiv …, 2024 - arxiv.org
Humans perceive the world through multisensory integration, blending the information of
different modalities to adapt their behavior. Contrastive learning offers an appealing solution …

Is AI fun? HumorDB: a curated dataset and benchmark to investigate graphical humor

V Jain, FSA Feitosa, G Kreiman - arxiv preprint arxiv:2406.13564, 2024 - arxiv.org
Despite significant advancements in computer vision, understanding complex scenes,
particularly those involving humor, remains a substantial challenge. This paper introduces …

Quantifying Knowledge Distillation Using Partial Information Decomposition

P Dissanayake, F Hamman, B Halder… - arxiv preprint arxiv …, 2024 - arxiv.org
Knowledge distillation provides an effective method for deploying complex machine learning
models in resource-constrained environments. It typically involves training a smaller student …

Mixture of Multimodal Interaction Experts

H Yu, PP Liang, R Salakhutdinov… - UniReps: the First …, 2023 - openreview.net
Multimodal machine learning, which studies the information and interactions across various
input modalities, has made significant advancements in understanding the relationship …