Visiontasker: Mobile task automation using vision based ui understanding and llm task planning

Y Song, Y Bian, Y Tang, G Ma, Z Cai - Proceedings of the 37th Annual …, 2024 - dl.acm.org
Mobile task automation is an emerging field that leverages AI to streamline and optimize the
execution of routine tasks on mobile devices, thereby enhancing efficiency and productivity …

AutoTask: Executing Arbitrary Voice Commands by Exploring and Learning from Mobile GUI

L Pan, B Wang, C Yu, Y Chen, X Zhang… - arxiv preprint arxiv …, 2023 - arxiv.org
Voice command interfaces (VCIs) have gained increasing importance, enabling hands-free
and eyes-free interaction with digital devices. However, the inherent complexity in …

Navigating Interfaces with AI for Enhanced User Interaction

Y Song, Y Bian, Y Tang, Z Cai - arxiv preprint arxiv:2312.11190, 2023 - arxiv.org
This study introduces an innovative framework designed to automate tasks by interacting
with UIs through a sequential, human-like problem-solving approach. Our approach initially …

PromptRPA: Generating Robotic Process Automation on Smartphones from Textual Prompts

T Huang, C Yu, W Shi, Z Peng, D Yang, W Sun… - arxiv preprint arxiv …, 2024 - arxiv.org
Robotic Process Automation (RPA) offers a valuable solution for efficiently automating tasks
on the graphical user interface (GUI), by emulating human interactions, without modifying …

Test2VA: Reusing GUI Test Cases for Voice Assistant Features Development in Mobile Applications

G Weaver, X Qin - arxiv preprint arxiv:2407.18155, 2024 - arxiv.org
Voice Assistant (VA) in smartphones has become very popular with millions of users
nowadays. A key trend is the rise of custom VA embedding, which enables users to perform …

Building Python Application for Webmail Interfaces Navigation Using Voice Recognition Technology

M Alkhattali, M Dow, K Azwee… - International Journal of …, 2023 - papers.ssrn.com
Abstract Voice Recognition Technology (VRT) has played a crucial role in technology
development, finding extensive use in the development of humanitarian assistance …

iTutor: A Generative Tutorial System for Teaching the Elders to Use Smartphone Applications

R Zou, Z Ye, C Ye - Adjunct Proceedings of the 36th Annual ACM …, 2023 - dl.acm.org
We present iTutor, a generative tutorial system for promoting smartphone use proficiency
among elders. iTutor is unique because it can dynamically generate tutorials based on …

Prompt2Task: Automating UI Tasks on Smartphones from Textual Prompts

T Huang, C Yu, W Shi, Z Peng, D Yang, W Sun… - ACM Transactions on … - dl.acm.org
UI task automation enables efficient task execution by simulating human interactions with
graphical user interfaces (GUIs), without modifying the existing application code. However …