Mouse2Vec: Learning Reusable Semantic Representations of Mouse Behaviour

Document Title

Mouse2Vec: Learning Reusable Semantic Representations of Mouse Behaviour

Document Information

• Subject Area: Human-Computer Interaction and Behavior Modeling
• Keywords: Mouse Input, Representation Learning, Self-Supervised Learning, Data Augmentation, Behavior Retrieval, Transformer

Research Background and Problem

• Problem and Challenges: Current mouse behavior modeling often relies on complex handcrafted feature design or application-specific models. This approach not only requires expert knowledge but is also time-consuming and lacks generalizability.
• Significance: The mouse is a widely used input device in human-computer interaction, and mouse behavior contains rich information about user interaction goals, cognitive states, etc. Modeling this behavior is crucial for improving personalized interaction systems.
• Research Motivation: Although self-supervised learning methods have been widely applied in other domains (e.g., user interfaces, speech, artwork, etc.), there is no dedicated self-supervised modeling method for mouse behavior learning.

Solution

• Proposed Method: Mouse2Vec is the first self-supervised method for mouse behavior learning. Its core is a Transformer-based encoder-decoder architecture:
  • The encoder learns embeddings of mouse trajectories;
  • The decoder reconstructs the input and identifies mouse click events.
• Innovations:
  • Simultaneously encodes the temporal and frequency domains (amplitude and phase) of mouse trajectories, as well as event information (clicks and movements);
  • Proposes a multi-task training strategy (input reconstruction + mouse event detection) to support broad application scenarios.
• Key Techniques and Steps:
  • Data Preprocessing: Mouse trajectories are normalized, resampled to 20Hz, and segmented;
  • Model Architecture: Uses a Transformer to process temporal and frequency information;
  • Multi-Task Training: Combines input reconstruction and click event detection tasks;
  • Pretrained on public datasets (Bufalo and EMAKI) to enhance model generalization.

Research Outcomes

• Specific Results:
  • Representations generated by Mouse2Vec capture semantic features and latent relationships of mouse behavior, supporting human-interpretable semantic analysis;
  • Effectively retrieves mouse trajectories with similar interaction characteristics.
• Advantages:
  • Compared to traditional methods based on handcrafted features, the model significantly reduces the time required for feature design;
  • The self-supervised approach eliminates the need for manual labels, reducing data annotation costs.
• Experimental Evaluation:
  • Achieved consistent performance improvements across three downstream tasks (interaction task recognition, next-step activity prediction, user identity recognition) and multiple datasets:
    • For example, interaction task recognition accuracy improved by 6.79% through data augmentation;
    • By directly using the pretrained model as a feature extractor, performance surpassed traditional handcrafted feature-based baseline methods.
  • Fine-tuning further enhanced model adaptability and performance.
• Limitations and Future Directions:
  • Currently focuses only on mouse behavior, without addressing other input devices (e.g., keyboard) or multimodal modeling;
  • Exploring the interaction between mouse behavior and user interface representations could further improve behavior modeling;
  • The release and use of more public datasets could further enhance model performance.

This research demonstrates the potential of self-supervised learning in the field of user behavior modeling, with significant practical application value, particularly in the design of intelligent user interfaces.