Language identification (" LI") is the problem of determining the natural language that a
document or part thereof is written in. Automatic LI has been extensively researched for over …

On Achieving and Evaluating Language-Independence in NLP Page 1 Linguistic Issues in
Language Technology LiLT Submitted, October 2011 On Achieving and Evaluating …

Abstract Language identification is the task of identifying the language a given document is
written in. This paper describes a detailed examination of what models perform best under …

In this paper we consider the problem of labeling the languages of words in mixed-language
documents. This problem is approached in a weakly supervised fashion, as a sequence …

We show that transductive (cross-domain) learning is an important consideration in building
a general-purpose language identification system, and develop a feature selection method …

Word-level language detection is necessary for analyzing code-switched text, where
multiple languages could be mixed within a sentence. Existing models are restricted to code …

Social media services such as Twitter offer an immense volume of real-world linguistic data.
We explore the use of Twitter to obtain authentic user-generated text in low-resource …

Abstract Language identification, as the task of determining the language a given text is
written in, has progressed substantially in recent decades. However, three main issues …

This paper presents a language identification algorithm using cosine similarity against a
filtered and weighted subset of the most frequent n-grams in training data with optional inter …

As the Internet and World Wide Web have continued to gain widespread adoption, the
linguistic diversity represented has also been growing. Simultaneously the field of …