Modeling user interfaces (UIs) from visual information allows systems to make inferences
about the functionality and semantics needed to support use cases in accessibility, app …

Graphical User Interface (GUI) has become one of the most significant parts of mobile
applications (apps). It is a direct bridge between mobile apps and end users, which directly …

Machine learning models have been trained to predict semantic information about user
interfaces (UIs) to make apps more accessible, easier to test, and to automate. Currently …

Mobile apps have become indispensable for accessing and participating in various
environments, especially for low-vision users. Users with visual impairments can use screen …

The emergent large language/multimodal models facilitate the evolution of mobile agents,
especially in mobile UI task automation. However, existing evaluation approaches, which …

Large language models (LLMs) struggle to consistently generate UI code that compiles and
produces visually relevant designs. Existing approaches to improve generation rely on …

Multimodal Vision-Language Models (VLMs) enable powerful applications from their fused
understanding of images and language, but many perform poorly on UI tasks due to the lack …

Vector graphics are an industry-standard way to represent and share visual designs.
Designers frequently source and incorporate styles from existing designs into their work …

Many digital wellbeing tools help users monitor and control social media use on their
smartphones by tracking and setting limits on their usage time. Tracking is typically done at …

Prior work in programming-by-demonstration (PBD) has explored ways to enable end-users
to create custom automation without needing to write code. We propose a new end-user …