Most existing benchmarks for grounding language in interactive environments either lack
realistic linguistic elements, or prove difficult to scale up due to substantial human …

Much of the previous work towards digital agents for graphical user interfaces (GUIs) has
relied on text-based representations (derived from HTML or other structured data sources) …

Large language models (LLMs) have shown exceptional performance on a variety of natural
language tasks. Yet, their capabilities for HTML understanding--ie, parsing the raw HTML of …

Modeling user interfaces (UIs) from visual information allows systems to make inferences
about the functionality and semantics needed to support use cases in accessibility, app …

Mobile apps have become indispensable for accessing and participating in various
environments, especially for low-vision users. Users with visual impairments can use screen …

Large language models (LLMs) have demonstrated remarkable capabilities in performing a
range of instruction-following tasks in few and zero-shot settings. However, teaching LLMs to …

Most instance perception and image understanding solutions focus mainly on natural
images. However, applications for synthetic images, and more specifically, images of …

With advances in generative AI, there is increasing work towards creating autonomous
agents that can manage daily tasks by operating user interfaces (UIs). While prior research …

Building autonomous agents to interact with the world lies at the core of artificial intelligence
(AI). This thesis introduces" language agents'', a new category of agents that utilize large …

A noteworthy omission in the development process of common NLP models is the lack of
interactive components. While common downstream applications of large language models …