Agents capable of carrying out general tasks on a computer can improve efficiency and
productivity by automating repetitive tasks and assisting in complex problem-solving. Ideally …

Most existing benchmarks for grounding language in interactive environments either lack
realistic linguistic elements, or prove difficult to scale up due to substantial human …

Since the advent of personal computing devices, intelligent personal assistants (IPAs) have
been one of the key technologies that researchers and engineers have focused on, aiming …

Conversational agents show the promise to allow users to interact with mobile devices using
language. However, to perform diverse UI tasks with natural language, developers typically …

Large language models (LLMs) have shown exceptional performance on a variety of natural
language tasks. Yet, their capabilities for HTML understanding--ie, parsing the raw HTML of …

Abstract Vision-language navigation (VLN), in which an agent follows language instruction
in a visual environment, has been studied under the premise that the input command is fully …

Representing the semantics of GUI screens and components is crucial to data-driven
computational methods for modeling user-GUI interactions and mining GUI designs. Existing …

We propose the problem of conversational web navigation, where a digital agent controls a
web browser and follows user instructions to solve real-world tasks in a multi-turn dialogue …

Abstract Graphical User Interface (GUI) automation holds significant promise for assisting
users with complex tasks thereby boosting human productivity. Existing works leveraging …

Task-oriented dialogue (TOD) systems have been widely used by mobile phone intelligent
assistants to accomplish tasks such as calendar scheduling or hotel reservation. Current …