Automatically Generating and Improving Voice Command Interface from Operation Sequences on Smartphones

Title of the Paper

Automatically Generating and Improving Voice Command Interface from Operation Sequences on Smartphones

Paper Information

• Subject Area: Voice Interaction and User Interface Design
• Keywords: Voice command interface, user operation sequences, natural language understanding, task automation, conversational systems, semantic learning, programming by demonstration, smartphone interaction

Research Background and Problem Statement

• Problems and Challenges:

  • Automating smartphone tasks (e.g., video calls) using voice commands can significantly enhance mobile application interactivity, but creating task-oriented voice user interfaces (VUI) for multiple applications is highly challenging.
  • Developers need to collect extensive corpora, create handwritten rules, or train machine learning models, resulting in a significant increase in development workload as the number of tasks grows.
  • Current commercial voice assistants support only limited tasks, while users, especially non-programmers, have a strong demand for voice interfaces that allow creating custom tasks.

• Significance:

  • Task automation and voice interaction can substantially reduce user operational burden, particularly for older users, greatly improving device usability and convenience.

• Motivation and Related Work:

  • Existing task configuration and user command understanding methods require extensive manual annotation and fail to adapt to the dynamic nature of natural language commands.
  • Programming by Demonstration (PBD) systems have high implementation costs and require additional input to extract parameterized operations.

Proposed Solution

• Proposed Approach:

  • Developed an automated system named AutoVCI to generate voice command interfaces from smartphone operation sequences.
  • AutoVCI converts users' touchscreen operation sequences into task semantics and automatically generates voice command-supported interfaces.
  • In cases of ambiguity in voice commands, AutoVCI resolves issues through supplementary dialogues designed to minimize user effort.

• Innovations:

  • Does not rely on corpora, handwritten rules, or pre-trained models, utilizing only operation sequences and runtime GUI data for semantic understanding and task mapping.
  • Features self-improvement capabilities: dynamically enhances semantic understanding through accumulated historical interaction data.
  • Introduced an information entropy optimization method to minimize the burden of additional user dialogue rounds.

• Implementation Steps:

  1. Collect user operation sequences and generate task semantic vectors (e.g., word embeddings).
  2. Identify user intent through direct semantic matching and determine task parameters from runtime GUI data.
  3. Handle ambiguities and confirm tasks via a supplementary dialogue module.
  4. Accumulate semantic data based on user interactions to optimize subsequent command processing.

Research Outcomes

• Specific Achievements:

  • Implemented AutoVCI on Android devices, generating voice command interfaces supporting 45 tasks across 11 applications.
  • Provided an automatic operation sequence collection tool, significantly simplifying interface design work.

• Comparison with Existing Solutions:

  • Compared to existing PBD systems, AutoVCI requires no additional input, greatly improving scalability.
  • Avoids semantic dynamic adaptation issues found in traditional rule-based or static NLP systems for command understanding.

• Experiments and Evaluation Results:

  • User Testing:
    • Phase 1 user study (N=16) showed users could automate tasks with a 98.4% success rate using AutoVCI. Semantic accumulation significantly reduced user interaction burden, with additional dialogue rounds decreasing from an average of 2.1 per command to 0.7.
    • Phase 2 online user study (N=67) further validated the effectiveness of semantic accumulation. Over 70% of commands were directly triggered, with additional dialogue rounds averaging only 0.4.
  • Offline Evaluation:
    • Demonstrated the superiority of the system's semantic understanding strategy, which combines NLP algorithms and GUI-encoded interaction semantics.

• Limitations and Future Directions:

  • The semantic accumulation speed during the cold start phase has not been fully optimized. Further research is needed to accelerate this process.
  • Additional dialogues could be further reduced by employing more advanced NLP techniques to optimize confirmation and parameter-related questions.
  • User experience in non-laboratory scenarios requires further improvement, such as evaluating user tolerance for supplementary dialogues.
  • Application version updates may lead to task execution failures; future work could explore better task maintenance and semantic relationship management solutions.