Tokenization is widely used in large language models because it significantly improves
performance. However, tokenization imposes several disadvantages, such as performance …

Language models typically tokenize text into subwords, using a deterministic, hand-
engineered heuristic of combining characters into longer surface-level strings such as' ing'or …

Tokenizers serve as crucial interfaces between models and linguistic data, substantially
influencing the efficacy and precision of large language models (LLMs). Traditional …

Current state-of-the-art models for natural language understanding require a preprocessing
step to convert raw text into discrete tokens. This process known as tokenization relies on a …

Static subword tokenization algorithms have been an essential component of recent works
on language modeling. However, their static nature results in important flaws that degrade …

Previous work has considered token overlap, or even similarity of token distributions, as
predictors for multilinguality and cross-lingual knowledge transfer in language models …

Text summarization is a significant topic in natural language processing. Tokenization
approaches are important in this regard as they underpin text recognition and processing …

With the use of Cross Linguistic Information Retrieval (CLIR) technology, users can ask a
question in one language and get answers to their request in a different language. For Tamil …

I wrote this text as a paper for the Digital Culture Seminar of the Digital Humanities faculty of
the University of Pisa, taking the opportunity to deepen and complete in a personal research …