Various forms of artificial intelligence (AI) applications are being deployed and used in many
healthcare systems. As the use of these applications increases, we are learning the failures …

In this work, we survey a breadth of literature that has revealed the limitations of
predominant practices for dataset collection and use in the field of machine learning. We …

We identify label errors in the test sets of 10 of the most commonly-used computer vision,
natural language, and audio datasets, and subsequently study the potential for these label …

With the growing processing power of computing systems and the increasing availability of
massive datasets, machine learning algorithms have led to major breakthroughs in many …

In this paper we investigate problematic practices and consequences of large scale vision
datasets (LSVDs). We examine broad issues such as the question of consent and justice as …

Yes, and no. We ask whether recent progress on the ImageNet classification benchmark
continues to represent meaningful generalization, or whether the community has started to …

Benchmark datasets play a central role in the organization of machine learning research.
They coordinate researchers around shared research problems and serve as a measure of …

Although self-/un-supervised methods have led to rapid progress in visual representation
learning, these methods generally treat objects and scenes using the same lens. In this …

We develop a methodology for assessing the robustness of models to subpopulation shift---
specifically, their ability to generalize to novel data subpopulations that were not observed …

ImageNet has been the most popular image classification benchmark, but it is also the one
with a significant level of label noise. Recent studies have shown that many samples contain …