HelpViz: Automatic Generation of Contextual Visual Mobile Tutorials from Text-Based Instructions

Document Title

HelpViz: Automatic Generation of Contextual Visual Mobile Tutorials from Text-Based Instructions

Document Information

• Research Area: Human-Computer Interaction and Automatic Generation of Mobile Application Tutorials
• Keywords: Mobile tutorials, automatic generation, contextual help, deep learning, graphical tutorials, user interaction design, simulated execution, tutorial generation, Android applications, accessibility

Research Background and Problem

• Challenges Identified by Authors: Mobile application tutorials are typically created by professionals, which is costly and time-consuming. Additionally, these tutorials need constant updates to align with software version changes. Many online tutorials exist in text format, requiring users to manually match instructions with device interfaces, which is time-consuming and prone to errors.
• Significance: Automating tutorial generation can reduce costs and quickly adapt tutorials following software updates. This helps mobile users navigate applications more effectively and reduces confusion.
• Motivation and Related Work: Previous studies have explored tutorial creation through expert demonstrations or crowdsourcing, but human involvement is still required. Literature reviews indicate that visual tutorials enhance comprehension and usability, but existing technologies rely on manual creation or semi-automatic generation, failing to achieve full automation.

Solution

• Methods and Approach:
  1. Text Parsing: Utilize pre-trained deep learning models to parse text instructions and extract actions and target objects.
  2. Step-by-Step Instruction Segmentation: Decompose instructions into multiple steps based on parsing results.
  3. Simulated Execution of Instructions: Automatically execute instructions in an array of Android emulators to capture screenshots, video clips, and metadata.
  4. Tutorial Synthesis: Combine text instructions with generated visual assets (e.g., screenshots and videos) to create interactive graphical tutorials.
  5. Real-Time Contextual Service: Dynamically display tutorials on mobile devices based on user actions, highlighting the current step being executed.
• Innovations:
  1. Parse text instructions using deep models to transform online text into graphical tutorials.
  2. Optimize the parsing model with "beam search" to provide multiple alternatives for potential errors.
  3. Employ "look-ahead" techniques to address mismatches between instructions and emulator interfaces, maximizing execution success rates.
• Implementation Steps and Key Technologies:
  1. Pre-trained instruction parsing models for action sequence generation.
  2. Batch execution of instructions across multiple emulators, handling screen loading issues with dynamic element waiting and scrolling mechanisms.
  3. Combine screenshots, videos, and metadata during tutorial synthesis to present operation interfaces in a multi-view format.

Research Outcomes

• Specific Results:
  1. Proposed the first complete pipeline for converting text instructions into visual tutorials automatically.
  2. Developed a comprehensive system from text parsing to emulator execution and context-sensitive tutorial presentation.
  3. User testing results showed a preference for HelpViz graphical tutorials, particularly in features like auto-scrolling and step highlighting.
• Advantages:
  1. Compared to traditional text tutorials, HelpViz-generated visual tutorials are easier to understand, supported by screenshots, and better suited to mobile device user needs.
  2. Automatically generated tutorials reduce reliance on professional expertise and address challenges posed by software version updates.
• Experimental and Evaluation Results:
  1. Pipeline Performance Evaluation: The system parsed and batch-converted 187 multi-step instructions from the PixelHelp dataset, achieving a completion rate of 80.9%, significantly higher than the baseline (67.1%).
  2. User Study: Ten participants compared two tutorial formats, favoring HelpViz, with subjective ratings significantly higher than traditional text tutorials.
  3. Technical Validation: "Beam search" and "look-ahead" methods significantly improved parsing accuracy and execution success rates.
• Limitations and Future Directions:
  1. Tutorial generation failures are primarily caused by severe mismatches between instruction text and application interfaces or unsupported emulator environments.
  2. Does not support complex workflows such as conditional branches or loops, requiring further expansion of parsing model capabilities.
  3. Optimize tutorials for different devices and versions to address Android fragmentation issues.
  4. Explore possibilities for adapting desktop software workflows and optimizing tutorial presentation for specific user groups, such as adding voice guidance for visually impaired users.

Through the research on HelpViz, the authors demonstrated the feasibility of large-scale automated generation of visual tutorials, providing higher-quality interactive help resources for mobile users while reducing tutorial maintenance costs. This novel approach offers strong support and inspiration for future automation in human-computer interaction tutorial processes.