Foundation and large language models: fundamentals, challenges, opportunities, and social impacts

D Myers, R Mohawesh, VI Chellaboina, AL Sathvik… - Cluster …, 2024‏ - Springer
Abstract Foundation and Large Language Models (FLLMs) are models that are trained using
a massive amount of data with the intent to perform a variety of downstream tasks. FLLMs …

Ammus: A survey of transformer-based pretrained models in natural language processing

KS Kalyan, A Rajasekharan, S Sangeetha - arxiv preprint arxiv …, 2021‏ - arxiv.org
Transformer-based pretrained language models (T-PTLMs) have achieved great success in
almost every NLP task. The evolution of these models started with GPT and BERT. These …

NusaCrowd: Open source initiative for Indonesian NLP resources

S Cahyawijaya, H Lovenia, AF Aji, GI Winata… - arxiv preprint arxiv …, 2022‏ - arxiv.org
We present NusaCrowd, a collaborative initiative to collect and unify existing resources for
Indonesian languages, including opening access to previously non-public resources …

Aya model: An instruction finetuned open-access multilingual language model

A Üstün, V Aryabumi, ZX Yong, WY Ko… - arxiv preprint arxiv …, 2024‏ - arxiv.org
Recent breakthroughs in large language models (LLMs) have centered around a handful of
data-rich languages. What does it take to broaden access to breakthroughs beyond first …

A survey of text representation and embedding techniques in nlp

R Patil, S Boit, V Gudivada, J Nandigam - IEEE Access, 2023‏ - ieeexplore.ieee.org
Natural Language Processing (NLP) is a research field where a language in consideration
is processed to understand its syntactic, semantic, and sentimental aspects. The …

End-to-end transformer-based models in textual-based NLP

A Rahali, MA Akhloufi - Ai, 2023‏ - mdpi.com
Transformer architectures are highly expressive because they use self-attention
mechanisms to encode long-range dependencies in the input sequences. In this paper, we …

One country, 700+ languages: NLP challenges for underrepresented languages and dialects in Indonesia

AF Aji, GI Winata, F Koto, S Cahyawijaya… - arxiv preprint arxiv …, 2022‏ - arxiv.org
NLP research is impeded by a lack of resources and awareness of the challenges presented
by underrepresented languages and dialects. Focusing on the languages spoken in …

Bactrian-x: Multilingual replicable instruction-following models with low-rank adaptation

H Li, F Koto, M Wu, AF Aji, T Baldwin - arxiv preprint arxiv:2305.15011, 2023‏ - arxiv.org
Instruction tuning has shown great promise in improving the performance of large language
models. However, research on multilingual instruction tuning has been limited due to the …

Indobertweet: A pretrained language model for indonesian twitter with effective domain-specific vocabulary initialization

F Koto, JH Lau, T Baldwin - arxiv preprint arxiv:2109.04607, 2021‏ - arxiv.org
We present IndoBERTweet, the first large-scale pretrained model for Indonesian Twitter that
is trained by extending a monolingually-trained Indonesian BERT model with additive …

NusaX: Multilingual parallel sentiment dataset for 10 Indonesian local languages

GI Winata, AF Aji, S Cahyawijaya, R Mahendra… - arxiv preprint arxiv …, 2022‏ - arxiv.org
Natural language processing (NLP) has a significant impact on society via technologies
such as machine translation and search engines. Despite its success, NLP technology is …