GUIs have long been central to human-computer interaction, providing an intuitive and
visually-driven way to access and interact with digital systems. The advent of LLMs …

Current Large Language Models (LLMs) are not only limited to some maximum context
length, but also are not able to robustly consume long inputs. To address these limitations …

Can advanced multi-modal models effectively tackle complex web-based tasks? Such tasks
are often found on crowdsourcing platforms, where crowdworkers engage in challenging …

Instruction-based computer control agents (CCAs) execute complex action sequences on
personal computers or mobile devices to fulfill tasks using the same graphical user …

The rapid advancement of neural language models has sparked a new surge of intelligent
agent research. Unlike traditional agents, large language model-based agents (LLM agents) …

Graphical User Interface (GUI) agents, powered by Large Foundation Models, have
emerged as a transformative approach to automating human-computer interaction. These …

Increasingly many AI systems can plan and execute interactions in open-ended
environments, such as making phone calls or buying online goods. As developers grow the …

The rapid advancement of neural language models has sparked a new surge of intelligent
agent research. Unlike traditional agents, large language model-based agents (LLM agents) …

As training datasets grow larger, we aspire to develop models that generalize well to any
diverse test distribution, even if the latter deviates significantly from the training data. Various …

The dream to create AI assistants as capable and versatile as the fictional JARVIS from Iron
Man has long captivated imaginations. With the evolution of (multimodal) large language …