App-Based Task Shortcuts for Virtual Assistants

Title of the Paper

App-Based Task Shortcuts for Virtual Assistants

Paper Information

• Domain: Human-Computer Interaction and Virtual Assistant Technology
• Keywords: Virtual Assistant, Mobile Applications, Interaction Mining, Task Shortcuts, User Interface, Intent Modeling, Human-Computer Interaction, Intelligent Automation, Deep Linking, Task Matching
• Conference: The 34th Annual ACM Symposium on User Interface Software and Technology (UIST ‘21)
• Publication Date: October 2021
• DOI: https://doi.org/10.1145/3472749.3474808

Research Background and Problem

• Problem or Challenge:

  • Current virtual assistant technologies (e.g., Google Assistant and Siri) have limited functionality, primarily relying on developers to manually expose application features to expand their skill sets (e.g., Android's App Actions or iOS Shortcuts).
  • This manual configuration requires additional development effort from developers, and many applications lack mechanisms for automatic support of specific tasks.
  • While virtual assistants can support certain tasks through existing programmatic configurations, this approach does not cover a wide range of user tasks and involves a complex implementation process.

• Significance:

  • Deep integration between virtual assistants and applications can significantly simplify how end users complete tasks.
  • Automatically expanding virtual assistants' support for more user tasks can enhance user experience and promote technological inclusivity.

• Motivation and Related Work:

  • Current solutions (e.g., deep linking and task automation based on programming by demonstration (PBD)) require developers to make additional efforts to expose functionalities, which does not directly benefit users.
  • The authors explore "interaction mining" to analyze user interfaces and interaction logs of mobile applications, aiming to automatically generate task shortcuts, reduce developer intervention, and address the limitations of current virtual assistant technologies.

Solution

• Proposed System: The savant system

  • This system can automatically map user-described tasks in natural language (e.g., "send money to Joe") to corresponding interfaces in mobile applications, thereby generating task shortcuts.
  • It uses semantic information from user interfaces and Google Play metadata for task matching and employs intent modeling to parse parameters within tasks.

• Innovations:

  • Automated Shortcut Generation: The savant system can automatically generate task shortcuts usable by virtual assistants without additional developer involvement.
  • Semantic Matching: Matches UI elements (e.g., buttons, icons) with task descriptions semantically, while leveraging app store descriptions to improve search results.
  • Support for Unlabeled Datasets: Utilizes unsupervised learning on interaction logs to find application interfaces that best match user tasks.
  • Task Filling and Automated Navigation: Implements slot filling using Dialogflow to automatically input task parameters into target application UI elements, supporting automated interaction playback.

• Implementation Steps and Techniques:

  1. Screen Representation Calculation for Shortcut Generation: Extracts semantic information from UI elements in interaction logs to generate screen semantic representations.
  2. Task Shortcut Generation: Uses cloud search services (AWS CloudSearch) to match user tasks with screen semantic representations.
  3. Shortcut Usage: Leverages Android's system-level permissions to launch application interfaces relevant to tasks and replays interaction paths when necessary.
  4. Slot Filling and Intent Modeling: Parses user task descriptions for intents and entities using Dialogflow and automatically fills UI component values.
  5. Infrastructure: Uses Python scripts for data preprocessing and Kotlin for Android application and system permission interactions.

Research Outcomes

• Specific Results:

  • Experiments validated that the savant system can effectively match user tasks with relevant application interfaces, providing support for tasks that virtual assistants cannot directly complete.
  • By generating task shortcuts for 66,000 UI screens, savant achieved an average precision of 70.1% for the top three shortcuts related to user tasks.

• Experiments and Evaluation:

  • Task Identification Study:
    • Conducted a user survey with 24 participants to define a set of 20 common task categories (e.g., sending messages, ordering food, reading).
    • Selected the most common tasks within each category as representative tasks.
  • Task Relevance Testing:
    • Evaluated the top-3 search results for representative tasks, with the system achieving 100% accuracy in five task types (e.g., "send money," "take a photo").

• Comparative Advantages:

  • Compared to manually configured deep linking methods, the savant system significantly reduces developer effort.
  • Unlike traditional task automation systems such as Sugilite and PUMICE, savant focuses more on discovering task shortcuts rather than broadly learning interaction workflows.

• Limitations and Future Directions:

  • For certain tasks (e.g., "post a photo"), the relevance of results was lower, indicating that keyword matching algorithms need improvement to better distinguish between verbs and nouns.
  • The current system is dependent on the Android ecosystem; future work should expand to other platforms (e.g., iOS) and devices such as smart TVs.
  • Privacy and security considerations need further attention, particularly in handling data traces generated during user-application interactions, such as the automatic cleaning of personally identifiable information (PII).