We introduce a set of image transformations that can be used as corruptions to evaluate the
robustness of models as well as data augmentation mechanisms for training neural …

Abstract Machine learning models have been found to learn shortcuts---unintended decision
rules that are unable to generalize---undermining models' reliability. Previous works address …

We present a conceptual framework, datamodeling, for analyzing the behavior of a model
class in terms of the training data. For any fixed" target" example $ x $, training set $ S $, and …

Deep neural networks can be unreliable in the real world especially when they heavily use
{\it spurious} features for their predictions. Focusing on image classifications, we define {\it …

Vision models often fail systematically on groups of data that share common semantic
characteristics (eg, rare objects or unusual scenes), but identifying these failure modes is a …

With the emergence of foundation models (FMs) that are trained on large amounts of data at
scale and adaptable to a wide range of downstream applications, AI is experiencing a …

Interpretable AI tools are often motivated by the goal of understanding model behavior in out-
of-distribution (OOD) contexts. Despite the attention this area of study receives, there are …

We study the problem of (learning) algorithm comparison, where the goal is to find
differences between models trained with two different learning algorithms. We begin by …

Recent multi-modal contrastive learning models have demonstrated the ability to learn an
embedding space suitable for building strong vision classifiers, by leveraging the rich …

Distribution shift is a major source of failure for machine learning models. However,
evaluating model reliability under distribution shift can be challenging, especially since it …