Large Language Models (LLMs) have demonstrated remarkable capabilities in important
tasks such as natural language understanding and language generation, and thus have the …

Humans generally acquire new skills without compromising the old; however, the opposite
holds for Large Language Models (LLMs), eg, from LLaMA to CodeLLaMA. To this end, we …

Current pre-trained language models (PLM) are typically trained with static data, ignoring
that in real-world scenarios, streaming data of various sources may continuously grow. This …

Scaling transformers has led to significant breakthroughs in many domains, leading to a
paradigm in which larger versions of existing models are trained and released on a periodic …

Training large models from scratch usually costs a substantial amount of resources. Towards
this problem, recent studies such as bert2BERT and LiGO have reused small pretrained …

Recent explorations of large-scale pre-trained language models (PLMs) have revealed the
power of PLMs with huge amounts of parameters, setting off a wave of training ever-larger …

Weight initialization plays an important role in neural network training. Widely used
initialization methods are proposed and evaluated for networks that are trained from scratch …

Large language models (LLMs) have achieved remarkable success in NLP and multimodal
tasks. Despite these successes, their development faces two main challenges:(i) high …

Quantization is of significance for compressing the over-parameterized deep neural models
and deploying them on resource-limited devices. Fixed-precision quantization suffers from …

Currently, most applications in the Industrial Internet of Things (IIoT) still rely on CNN-based
neural networks. Although Transformer-based large models (LMs), including language …