Over the past decade, deep learning has proven to be a highly effective tool for learning
meaningful features from raw data. However, it remains an open question how deep …

Large-scale foundation models have demonstrated exceptional performance in language
and vision tasks. However, the numerous dense matrix-vector operations involved in these …

Lookahead is a popular stochastic optimizer that can accelerate the training process of deep
neural networks. However, the solutions found by Lookahead often generalize worse than …

We study deep neural networks for the multi-label classification (MLab) task through the lens
of neural collapse (NC). Previous works have been restricted to the multi-class classification …

Foundation models pre-trained on large-scale data have been widely witnessed to achieve
success in various natural imaging downstream tasks. Parameter-efficient fine-tuning (PEFT) …

We propose an empirical approach centered on the spectral dynamics of weights--the
behavior of singular values and vectors during optimization--to unify and clarify several …

Feature-learning deep nets progressively collapse data to a regular low-dimensional
geometry. How this phenomenon emerges from collective action of nonlinearity, noise …

This paper investigates efficient deep neural networks (DNNs) to replace dense
unstructured weight matrices with structured ones that possess desired properties. The …

In this paper, we study equality-type Clarke subdifferential chain rules of matrix factorization
and factorization machine. Specifically, we show for these problems that provided the latent …

Overparameterized models have proven to be powerful tools for solving various machine
learning tasks. However, overparameterization often leads to a substantial increase in …