We present a method to learn compositional multi-object dynamics models from image
observations based on implicit object encoders, Neural Radiance Fields (NeRFs), and …

We propose to forecast future hand-object interactions given an egocentric video. Instead of
predicting action labels or pixels, we directly predict the hand motion trajectory and the …

Understanding dynamics from visual observations is a challenging problem that requires
disentangling individual objects from the scene and learning their interactions. While recent …

In this work, we propose a unified framework, called Visual Reasoning with Differ-entiable
Physics (VRDP), that can jointly learn visual concepts and infer physics models of objects …

Abstract Research on Inverse Reinforcement Learning (IRL) from third-person videos has
shown encouraging results on removing the need for manual reward design for robotic …

Visual environments are structured, consisting of distinct objects or entities. These entities
have properties---visible or latent---that determine the manner in which they interact with one …

A prominent approach to visual Reinforcement Learning (RL) is to learn an internal state
representation using self-supervised methods, which has the potential benefit of improved …

While current vision algorithms excel at many challenging tasks, it is unclear how well they
understand the physical dynamics of real-world environments. Here we introduce Physion, a …

In order to enable the model to generalize to unseen “action-objects”(compositional action),
previous methods encode multiple pieces of information (ie, the appearance, position, and …

This work introduces Video Diffusion Transformer (VDT), which pioneers the use of
transformers in diffusion-based video generation. It features transformer blocks with …