Abstract Evaluation practices in natural language generation (NLG) have many known flaws,
but improved evaluation approaches are rarely widely adopted. This issue has become …

This article presents a review of signals used for measuring physiology and activity during
sleep and techniques for extracting information from these signals. We examine both clinical …

Machine learning (ML) depends on data to train and verify models. Very often, organizations
outsource processes related to data work (ie, generating and annotating data and …

Research in machine learning (ML) has argued that models trained on incomplete or biased
datasets can lead to discriminatory outputs. In this commentary, we propose moving the …

This article is a survey of methods for measuring agreement among corpus annotators. It
exposes the mathematics and underlying assumptions of agreement coefficients, covering …

This article describes the enrichment of the AnCora corpora of Spanish and Catalan (400 k
each) with coreference links between pronouns (including elliptical subjects and clitics), full …

We observe a severe under-reporting of the different kinds of errors that Natural Language
Generation systems make. This is a problem, because mistakes are an important indicator of …

The question of data reliability is of first importance to assess the quality of manually
annotated corpora. Although Cohen's κ is the prevailing reliability measure used in NLP …

In computational linguistics, a reliability measurement of 0.8 on some statistic such as κ is
widely thought to guarantee that hand-coded data is fit for purpose, with 0.67 to 0.8 …

In recent years, ransomware attacks have led to disastrous consequences for victims, not
just due to the payment ransom amount but also due to the recovery costs associated with …