Recent studies have demonstrated the efficiency of generative pretraining for English
natural language understanding. In this work, we extend this approach to multiple …

State-of-the-art natural language processing systems rely on supervision in the form of
annotated data to learn competent models. These models are generally trained on data in a …

Distributed word representations, or word vectors, have recently been applied to many tasks
in natural language processing, leading to state-of-the-art performance. A key ingredient to …

We present Unicoder, a universal language encoder that is insensitive to different
languages. Given an arbitrary NLP task, a model can be trained with Unicoder using training …

Tokenization is a fundamental preprocessing step for almost all NLP tasks. In this paper, we
propose efficient algorithms for the WordPiece tokenization used in BERT, from single-word …

The Handbook of Natural Language Processing, Second Edition presents practical tools
and techniques for implementing natural language processing in computer systems. Along …

In this work we focus on transferring supervision signals of natural language generation
(NLG) tasks between multiple languages. We propose to pretrain the encoder and the …

Text data are ubiquitous and play an essential role in big data applications. However, text
data are mostly unstructured. Transforming unstructured text into structured units (eg …

To proactively mitigate malware threats, cybersecurity tools, such as anti-virus and anti-
malware software, as well as firewalls, require frequent updates and proactive …

Recurrent neural network (RNN) has been broadly applied to natural language process
(NLP) problems. This kind of neural network is designed for modeling sequential data and …