Achieving human-like planning and control with multimodal observations in an open world is
a key milestone for more functional generalist agents. Existing approaches can handle …

Computational experiments have emerged as a valuable method for studying complex
systems, involving the algorithmization of counterfactuals. However, accurately representing …

Vision-language models (VLMs) have excelled in multimodal tasks, but adapting them to
embodied decision-making in open-world environments presents challenges. One critical …

We study building embodied agents for open-ended creative tasks. While existing methods
build instruction-following agents that can perform diverse open-ended tasks, none of them …

This paper presents OmniJARVIS, a novel Vision-Language-Action (VLA) model for open-
world instruction-following agents in Minecraft. Compared to prior works that either emit …

Recent studies have delved into constructing generalist agents for open-world environments
like Minecraft. Despite the encouraging results, existing efforts mainly focus on solving basic …

This paper studies the problem of learning an agent policy that can follow various forms of
instructions. Specifically, we focus on multi-modal instructions: the policy is expected to …

LMMs have shown impressive visual understanding capabilities, with the potential to be
applied in agents, which demand strong reasoning and planning abilities. Nevertheless …