A comprehensive survey on pretrained foundation models: A history from bert to chatgpt

C Zhou, Q Li, C Li, J Yu, Y Liu, G Wang… - International Journal of …, 2024‏ - Springer
Abstract Pretrained Foundation Models (PFMs) are regarded as the foundation for various
downstream tasks across different data modalities. A PFM (eg, BERT, ChatGPT, GPT-4) is …

Multimodal sentiment analysis: a survey of methods, trends, and challenges

R Das, TD Singh - ACM Computing Surveys, 2023‏ - dl.acm.org
Sentiment analysis has come long way since it was introduced as a natural language
processing task nearly 20 years ago. Sentiment analysis aims to extract the underlying …

Text classification via large language models

X Sun, X Li, J Li, F Wu, S Guo, T Zhang… - arxiv preprint arxiv …, 2023‏ - arxiv.org
Despite the remarkable success of large-scale Language Models (LLMs) such as GPT-3,
their performances still significantly underperform fine-tuned models in the task of text …

Scaling vision transformers to gigapixel images via hierarchical self-supervised learning

RJ Chen, C Chen, Y Li, TY Chen… - Proceedings of the …, 2022‏ - openaccess.thecvf.com
Abstract Vision Transformers (ViTs) and their multi-scale and hierarchical variations have
been successful at capturing image representations but their use has been generally …

A survey on text classification: From traditional to deep learning

Q Li, H Peng, J Li, C **a, R Yang, L Sun… - ACM Transactions on …, 2022‏ - dl.acm.org
Text classification is the most fundamental and essential task in natural language
processing. The last decade has seen a surge of research in this area due to the …

A general survey on attention mechanisms in deep learning

G Brauwers, F Frasincar - IEEE Transactions on Knowledge …, 2021‏ - ieeexplore.ieee.org
Attention is an important mechanism that can be employed for a variety of deep learning
models across many different domains and tasks. This survey provides an overview of the …

[HTML][HTML] Bidirectional convolutional recurrent neural network architecture with group-wise enhancement mechanism for text sentiment classification

A Onan - Journal of King Saud University-Computer and …, 2022‏ - Elsevier
Sentiment analysis has been a well-studied research direction in computational linguistics.
Deep neural network models, including convolutional neural networks (CNN) and recurrent …

A review on the attention mechanism of deep learning

Z Niu, G Zhong, H Yu - Neurocomputing, 2021‏ - Elsevier
Attention has arguably become one of the most important concepts in the deep learning
field. It is inspired by the biological systems of humans that tend to focus on the distinctive …

LongT5: Efficient text-to-text transformer for long sequences

M Guo, J Ainslie, D Uthus, S Ontanon, J Ni… - arxiv preprint arxiv …, 2021‏ - arxiv.org
Recent work has shown that either (1) increasing the input length or (2) increasing model
size can improve the performance of Transformer-based neural models. In this paper, we …

Rethinking attention with performers

K Choromanski, V Likhosherstov, D Dohan… - arxiv preprint arxiv …, 2020‏ - arxiv.org
We introduce Performers, Transformer architectures which can estimate regular (softmax)
full-rank-attention Transformers with provable accuracy, but using only linear (as opposed to …