Sparse expert models are a thirty-year old concept re-emerging as a popular architecture in
deep learning. This class of architecture encompasses Mixture-of-Experts, Switch …

Transfer learning has recently become the dominant paradigm of machine learning. Pre-
trained models fine-tuned for downstream tasks achieve better performance with fewer …

Sparse mixture of experts provides larger model capacity while requiring a constant
computational overhead. It employs the routing mechanism to distribute input tokens to the …

Sparse Mixtures of Experts (SMoE) scales model capacity without significant increases in
training and inference costs, but exhibits the following two issues:(1) Low expert activation …

Multilingual speech recognition for both monolingual and code-switching speech is a
challenging task. Recently, based on the Mixture of Experts (MoE), many works have made …

Abstract Sparsely Mixture of Experts (MoE) has received great interest due to its promising
scaling capability with affordable computational overhead. MoE models convert dense …

Multi-lingual speech recognition aims to distinguish linguistic expressions in different
languages and integrate acoustic processing simultaneously. In contrast, current …

Multi-Head Mixture-of-Experts (MH-MoE) demonstrates superior performance by using the
multi-head mechanism to collectively attend to information from various representation …

Sparse Mixture of Experts (SMoE) has become the key to unlocking unparalleled scalability
in deep learning. SMoE has the potential to exponentially increase parameter count while …

This study addresses the speech enhancement (SE) task within the causal inference
paradigm by modeling the noise presence as an intervention. Based on the potential …