In this work, we discuss building performant Multimodal Large Language Models (MLLMs).
In particular, we study the importance of various architecture components and data choices …

Understanding the parameter estimation of softmax gating Gaussian mixture of experts has
remained a long-standing open problem in the literature. It is mainly due to three …

Multi-scenario learning (MSL) enables a service provider to cater for users' fine-grained
demands by separating services for different user sectors, eg, by user's geographical region …

In this paper, we present DiT-MoE, a sparse version of the diffusion Transformer, that is
scalable and competitive with dense networks while exhibiting highly optimized inference …

Quantitative stock investment is a fundamental financial task that highly relies on accurate
prediction of market status and profitable investment decision making. Despite recent …

Mixture of experts (MoE) model is a statistical machine learning design that aggregates
multiple expert networks using a softmax gating function in order to form a more intricate and …

Abstract Sparsely gated Mixture-of-Expert (MoE) has demonstrated its effectiveness in
scaling up deep neural networks to an extreme scale. Despite that numerous efforts have …

Mixture-of-Experts (MoE) architectures face challenges such as high memory consumption
and redundancy in experts. Pruning MoE can reduce network weights while maintaining …

Sparse mixture of experts (SMoE) offers an appealing solution to scale up the model
complexity beyond the mean of increasing the network's depth or width. However, effective …

The sparse Mixture-of-Experts (Sparse-MoE) framework efficiently scales up model capacity
in various domains, such as natural language processing and vision. Sparse-MoEs select a …