Motion-Touch: Kinematic-based Adaptive Switch for Enhancing Virtual-Hand Selection with Target Prediction in AR/VR

Paper Title

Motion-Touch: Kinematic-based Adaptive Switch for Enhancing Virtual-Hand Selection with Target Prediction in AR/VR

Publication Info

• Topic area: Virtual-hand selection techniques in AR/VR with focus on speed-accuracy trade-offs.
• Keywords: Virtual hand, AR/VR interaction, target prediction, kinematics-based adaptive switch, machine learning, speed-accuracy trade-off, error mitigation, user experience, deep learning, interaction design.

Background and Problem

• Problem / challenge: Virtual-hand selection techniques face inherent speed–accuracy trade-offs, with current methods introducing latency, errors, or interaction disruptions. Predictive models offer promise but suffer from inherent inaccuracies, particularly in dynamic and complex environments.
• Significance: Addressing these limitations is critical for improving the usability and efficiency of AR/VR interfaces, enabling seamless and intuitive interactions in virtual environments.
• Motivation and related work: Previous methods include spatial coupling, dwell time, gesture-based triggering, and predictive models. While these approaches improve accuracy or speed, they introduce drawbacks such as latency, instability, or reliance on fixed motion assumptions. Machine learning-based prediction models have been explored but remain underutilized for virtual-hand interactions due to inherent errors and lack of integration with dynamic user behavior.

Solution

• Proposed approach: Motion-Touch, a virtual-hand selection technique combining a Transformer-based target prediction model with a Kinematic-Based Adaptive Switch (KBAS) to dynamically regulate trigger states based on user motion phases.
• Novelty:
  1. Integration of kinematic-based adaptive switching with machine learning for error mitigation.
  2. Use of a Transformer Encoder for real-time target prediction based on hand kinematics.
  3. Dynamic adjustment of thresholds for switching states based on target difficulty.
  4. Validation of the approach in realistic AR/VR scenarios with dense and cluttered environments.
• Procedure and key techniques:
  • Hand kinematics data collected via HMD sensors.
  • Transformer Encoder predicts fingertip position 10 frames ahead using features like position, direction, speed, and acceleration.
  • KBAS dynamically switches between non-triggerable (ballistic phase) and triggerable (corrective phase) states based on speed, acceleration, and directional deviation.
  • Empirical thresholds for KBAS calibrated using Otsu algorithm and adjusted dynamically for target size.

Results

• Concrete findings:
  • Motion-Touch achieved trigger times < 0.1s and error rates < 1%.
  • Outperformed Magic-Tap and Clicker in high-difficulty tasks (15.68% faster than Clicker).
  • Maintained low error rates (0.17% in 2D grid tasks; 0.87% in 3D shelf tasks).
• Advantage over baselines:
  • Faster trigger times and higher throughput compared to Magic-Tap and Clicker.
  • Comparable error rates to Clicker and Magic-Tap, with improved efficiency in challenging conditions.
  • Superior user experience ratings in attributes like efficiency, stimulation, and novelty.
• Experiments / evaluation:
  • Study One: Model training using hand motion data from 20 participants; validated feature selection and model performance (Transformer Encoder reduced prediction error to 47.1mm).
  • Study Two: Comparison of Motion-Touch with Magic-Tap and Clicker across nine Index of Difficulty (ID) tasks; evaluated metrics like task completion time, error rate, throughput, and trigger time.
  • Realistic scenarios: Tested Motion-Touch in dense 2D grid and cluttered 3D shelf environments with 12 participants; demonstrated high precision and fault tolerance.
• Limitations and future work:
  • Model processes each selection independently, lacking sequential prediction for continuous tasks.
  • Fixed sliding window size may limit adaptability; future work will explore dynamic window mechanisms.
  • Exclusively tested with right-handed participants; plans to include left-handed users and multimodal inputs like eye-gaze for enhanced prediction.

Summary

Motion-Touch introduces a kinematics-based adaptive switch integrated with machine learning to address the speed–accuracy trade-off in virtual-hand selection techniques for AR/VR. By dynamically regulating trigger states based on user motion phases, it mitigates prediction errors and achieves superior performance in terms of speed, accuracy, and user experience compared to existing techniques like Magic-Tap and Clicker. Experimental validation across diverse tasks and realistic scenarios demonstrates its efficiency, reliability, and applicability for complex environments. Future work aims to refine sequential prediction models, expand multimodal inputs, and explore broader interaction designs.