Multimodal machine learning is a vibrant multi-disciplinary research field that aims to design
computer agents with intelligent capabilities such as understanding, reasoning, and learning …

Learning multimodal representations involves integrating information from multiple
heterogeneous sources of data. It is a challenging yet crucial area with numerous real-world …

Modeling user interfaces (UIs) from visual information allows systems to make inferences
about the functionality and semantics needed to support use cases in accessibility, app …

The field of generative artificial intelligence has seen significant advancements in recent
years with the advent of large language models, which have shown impressive results in …

Accurately recognizing icon types in mobile applications is integral to many tasks, including
accessibility improvement, UI design search, and conversational agents. Existing research …

Machine learning models have been trained to predict semantic information about user
interfaces (UIs) to make apps more accessible, easier to test, and to automate. Currently …

Rapid GUI prototy** has evolved into a widely applied technique in early stages of
software development to facilitate the clarification and refinement of requirements …

Enabling machines to understand structured visuals like slides and user interfaces is
essential for making them accessible to people with disabilities. However, achieving such …

UI designers often correct false affordances and improve the discoverability of features when
users have trouble determining if elements are tappable. We contribute a novel system that …

All graphical user interfaces are comprised of one or more screens that may be shown to the
user depending on their interactions. Identifying different screens of an app and …