Large-scale Text-to-image Generation Models (LTGMs)(eg, DALL-E), self-supervised deep
learning models trained on a huge dataset, have demonstrated the capacity for generating …

Recent interest in design through the artificial intelligence (AI) lens is rapidly increasing.
Designers, as a special user group interacting with AI, have received more attention in the …

Visually-situated language is ubiquitous—sources range from textbooks with diagrams to
web pages with images and tables, to mobile apps with buttons and forms. Perhaps due to …

There is a growing interest in device-control systems that can interpret human natural
language instructions and execute them on a digital device by directly controlling its user …

Dark patterns are user interface design choices that benefit an online service by coercing,
steering, or deceiving users into making unintended and potentially harmful decisions. We …

Mobile User Interface Summarization generates succinct language descriptions of mobile
screens for conveying important contents and functionalities of the screen, which can be …

Modeling user interfaces (UIs) from visual information allows systems to make inferences
about the functionality and semantics needed to support use cases in accessibility, app …

Large language model (LLM) leads to a surge of autonomous GUI agents for smartphone,
which completes a task triggered by natural language through predicting a sequence of …

Conditional graphic layout generation, which automatically maps user constraints to high-
quality layouts, has attracted widespread attention today. Although recent works have …

Many accessibility features available on mobile platforms require applications (apps) to
provide complete and accurate metadata describing user interface (UI) components …