Large language models (LLMs) are gaining increasing popularity in both academia and
industry, owing to their unprecedented performance in various applications. As LLMs …

Benchmarks for large multimodal language models (MLMs) now serve to simultaneously
assess the general capabilities of models instead of evaluating for a specific capability. As a …

Deployed multimodal models can fail in ways that evaluators did not anticipate. In order to
find these failures before deployment, we introduce MultiMon, a system that automatically …

People are relying on AI agents to assist them with various tasks. The human must know
when to rely on the agent, collaborate with the agent, or ignore its suggestions. In this work …

Distribution shift is a major source of failure for machine learning models. However,
evaluating model reliability under distribution shift can be challenging, especially since it …

Large language models (LLMs) have achieved remarkable performance in various
evaluation benchmarks. However, concerns are raised about potential data contamination in …

Despite excellent average-case performance of many image classifiers, their performance
can substantially deteriorate on semantically coherent subgroups of the data that were …

The distribution of subpopulations is an important property hidden within a dataset.
Uncovering and analyzing the subpopulation distribution within datasets provides a …

Evaluation of large language models (LLMs) has raised great concerns in the community
due to the issue of data contamination. Existing work designed evaluation protocols using …

Recent works have shown that Large Language Models (LLMs) could empower traditional
neuro-symbolic models via programming capabilities to translate language into module …