Humans possess an extraordinary ability to create and utilize tools. With the advent of
foundation models, artificial intelligence systems have the potential to be equally adept in …

Abstract We introduce Mind2Web, the first dataset for develo** and evaluating generalist
agents for the web that can follow language instructions to complete complex tasks on any …

Agents capable of carrying out general tasks on a computer can improve efficiency and
productivity by automating repetitive tasks and assisting in complex problem-solving. Ideally …

With advances in generative AI, there is now potential for autonomous agents to manage
daily tasks via natural language commands. However, current agents are primarily created …

There is a growing interest in device-control systems that can interpret human natural
language instructions and execute them on a digital device by directly controlling its user …

AutoGen is an open-source framework that allows developers to build LLM applications via
multiple agents that can converse with each other to accomplish tasks. AutoGen agents are …

Since the advent of personal computing devices, intelligent personal assistants (IPAs) have
been one of the key technologies that researchers and engineers have focused on, aiming …

The recent development on large multimodal models (LMMs), especially GPT-4V (ision) and
Gemini, has been quickly expanding the capability boundaries of multimodal models …

Most existing benchmarks for grounding language in interactive environments either lack
realistic linguistic elements, or prove difficult to scale up due to substantial human …

While language models (LMs) have shown potential across a range of decision-making
tasks, their reliance on simple acting processes limits their broad deployment as …