Log2Motion: Biomechanical Motion Synthesis from Touch Logs

Paper Title

Log2Motion: Biomechanical Motion Synthesis from Touch Logs

Publication Info

• Topic area: Biomechanical motion synthesis for touch interaction analysis.
• Keywords: Biomechanical simulation, motion synthesis, touch logs, reinforcement learning, musculoskeletal modeling, human-computer interaction, ergonomics, usability, motor control, mobile devices.

Background and Problem

• Problem / challenge: Existing touch logs provide limited insight into the physical processes behind user interactions. Prior methods predict performance metrics but fail to model the biomechanical processes that generate interactions. Current biomechanical models are task-specific and disconnected from real-world applications.
• Significance: Understanding the physical processes behind touch interactions can inform UI design, ergonomics, and accessibility, reducing the need for costly user studies.
• Motivation and related work: Previous research has focused on log-based analysis, forward models, and biomechanical simulation, but these approaches lack integration with real-world applications or fail to model fine motor control. This paper addresses these gaps by introducing a method to synthesize user motion from touch logs.

Solution

• Proposed approach: Log2Motion, a method for synthesizing biomechanically plausible motion from touch logs using reinforcement learning and musculoskeletal forward simulation.
• Novelty:
  1. Formulation of motion synthesis as a Partially Observable Markov Decision Process (POMDP) for dexterous touch interactions.
  2. Integration of a software emulator with a physics-based biomechanical simulator to link real-world applications with motion synthesis.
  3. Development of a reward model for learning human-like movement policies.
• Procedure and key techniques:
  • Define motion synthesis as a POMDP with states, observations, actions, and rewards.
  • Integrate Android Emulator with MuJoCo physics engine for real-time simulation.
  • Use RL to train motor operators (e.g., tapping, swiping) with modular reward functions.
  • Model motor noise and effort to reflect human variability and perceived exertion.
  • Evaluate synthesized movements against human data for plausibility and accuracy.

Results

• Concrete findings:
  • Synthesized movements align with Fitts’ Law (R² > 0.96) and human motor behavior.
  • Error rates for tapping policies: 8.5% (accurate), 12.5% (normal), 42.5% (fast) on 4 mm buttons.
  • Muscle effort analysis shows trade-offs between peak and total effort for different policies.
  • Simulated trajectories closely match human motion capture data (median DTW distances: 1.43 cm for accurate, 1.29 cm for fast).
• Advantage over baselines:
  • Generates human-like motion trajectories with realistic velocity profiles and effort metrics.
  • Supports arbitrary interaction sequences and adapts to new postures with minimal fine-tuning.
  • Provides insights into ergonomics and performance not available from logs alone.
• Experiments / evaluation:
  • Comparison with human data for pointing and swiping tasks.
  • Validation against Fitts’ Law and empirical movement data.
  • Analysis of error rates, velocity profiles, and muscle effort across tapping and swiping policies.
  • Application to large-scale logs (Android-in-the-Wild dataset) to predict error rates, task duration, and effort.
• Limitations and future work:
  • Current model supports single-handed, index-finger interactions with flat-on-table devices.
  • Future work includes adding two-handed interactions, thumb-based input, and cognitive/perceptual modeling.
  • Potential for multi-objective optimization and automated RL for improved training.

Summary

Log2Motion introduces a novel method for synthesizing biomechanically plausible motion from touch logs, bridging the gap between interaction logs and the physical processes generating them. By integrating a software emulator with a biomechanical simulator and leveraging reinforcement learning, it produces human-like motion trajectories consistent with empirical data. The approach enables the estimation of performance metrics such as speed, accuracy, and effort, providing insights into ergonomics and usability. Applications include augmenting log data, evaluating accessibility, and exploring alternative designs. Future work aims to expand supported interactions and integrate cognitive and perceptual models.