Video2Action: Reducing Human Interactions in Action Annotation of App Tutorial Videos

Document Title

Video2Action: Reducing Human Interactions in Action Annotation of App Tutorial Videos

Document Information

• Subject Area: Human-Computer Interaction, Video Annotation, and User Behavior Prediction
• Keywords: App tutorial videos, user actions, deep learning, automated image processing, UI interaction, video scene segmentation

Research Background and Problem

• Identified Problems or Challenges:

  1. Annotating user actions in mobile app tutorial videos requires significant time and effort.
  2. Users find it difficult to perceive rapid actions and subtle UI changes in videos.
  3. Auditory information in videos presents language barriers and is unfriendly to hearing-impaired users.
  4. Existing methods, such as embedding API code or using external recording devices, can generate partial action data but are tools designed for developers and testers, making them unsuitable for non-technical users like video creators.

• Importance of the Problem:
  Tutorial videos have become an essential way for users to learn new features and techniques. However, the lack of supporting tools makes the annotation process cumbersome, resulting in many videos lacking key action annotations, which diminishes the effectiveness and usability of tutorials.

• Research Motivation and Related Work:

  • Existing research primarily focuses on video annotation in natural scenes or desktop software, with a lack of modeling methods for user actions in mobile UI tutorial videos.
  • The goal is to quickly and efficiently annotate actions and their locations, even for operations that cannot be directly labeled, while minimizing human intervention.

Solution

• Method or Solution:
  A lightweight, non-intrusive method called Video2Action is proposed, which leverages image processing and deep learning techniques to automatically extract user actions from videos.

• Innovations:

  1. Heuristic image processing methods are used to segment scenes in tutorial videos.
  2. A deep learning model predicts action locations, utilizing data augmentation and optimized loss functions to enhance model robustness.
  3. An interactive interface is provided for video creators to assist in action annotation.

• Implementation Steps and Key Techniques:

  1. Operation Scene Generation:
     • Brightness comparison algorithms calculate structural similarity (SSIM) between video frames to segment scenes with significant UI changes.
     • Similarity change patterns are used to identify three types of actions: Tap, Scroll, and Backward.
  2. Action Location Prediction:
     • A Region Proposal Network (RPN) detects potential clickable areas.
     • A combination of visual encoding and location prediction networks predicts the click location triggering UI changes.
     • Specific data augmentation methods, such as element swapping and deformation augmentation, are employed to expand the training dataset.
  3. Interface Implementation:
     • The generated scenes and predicted locations are integrated into an interactive user interface, enabling video creators to use them directly.

Research Outcomes

• Specific Results:

  • Video2Action achieved an F1 score of 81.67% for scene segmentation tasks and a Levenshtein score of 86.41% in automated experiments.
  • For action location prediction, the Top-1 accuracy reached 50.14%, and the Top-5 accuracy was 81.89%, outperforming existing methods.
  • User studies showed that Video2Action reduced annotation time by 85%, with feedback confirming its benefits for the video community.

• Comparative Advantages:

  • Compared to previous image processing and machine learning methods, Video2Action is better suited to the characteristics of mobile app interfaces, effectively extracting action information from video input alone.
  • The fully interactive interface significantly lowers the learning and usage barriers for creators.

• Experimental or Evaluation Results:

  • Automated evaluations demonstrated that the proposed method significantly outperforms seven baseline approaches in tutorial video scene segmentation and action prediction tasks.
  • User testing revealed that video creators significantly improved annotation speed using the tool and could easily correct inaccuracies.

• Limitations and Future Directions:

  • Limitations include:
    1. Errors in scene generation for UI videos with frequent animations or initial Backward actions.
    2. Limited ability to predict deep semantic understanding and content differences in complex scenes.
  • Future directions:
    1. Improve accuracy by integrating dynamic data such as UI transition animations.
    2. Extend the method to more advanced user actions, such as pinch-to-zoom and 3D interactions.
    3. Provide cross-platform support (e.g., action analysis for iOS and web tutorial videos).

In summary, Video2Action offers an innovative and effective solution to address key challenges in tutorial video annotation, demonstrating significant performance advantages and strong practical utility.