HandyCast: Phone-based Bimanual Input for Virtual Reality in Mobile and Space-Constrained Settings via Pose-and-Touch Transfer

Document Title

HandyCast: Phone-based Bimanual Input for Virtual Reality in Mobile and Space-Constrained Settings via Pose-and-Touch Transfer

Document Information

• Subject Area: Virtual Reality (VR) interaction technology, mobile controllers in space-constrained environments
• Keywords: Virtual Reality, VR input, interaction technology, 3D controllers, bimanual interaction, phone input, space constraints, six degrees of freedom tracking, amplification mapping, SteamVR

Research Background and Problem

• Problems or Challenges Identified by the Authors:

  1. Current VR systems are typically designed for standing interactions in large spaces, requiring users to be in physical environments with minimal obstacles, which cannot meet the needs of mobile and space-constrained scenarios such as interactions in cars, public transportation, or waiting areas.
  2. Using standard VR handheld controllers may require larger physical space, making them unsuitable for narrow environments.

• Why This Problem is Important:

  • As VR devices become more widespread, users need to utilize VR in mobile and space-limited scenarios, making it essential to design efficient interaction methods for these contexts.

• Research Motivation and Related Work:

  • The HandyCast method aims to achieve bimanual control via users' smartphones to address space-constrained challenges. Compared to traditional VR controllers, this study envisions using widely available smartphones to reduce hardware requirements without compromising interaction effectiveness.

Solution

• Proposed Method or Solution:

  • HandyCast is a smartphone-based input technique that integrates phone position, orientation, and touchscreen input to control two virtual hands through a "pose-and-touch transfer function."
  • Introduced a 10 degrees of freedom input mode combining 6D (3D position and 3D orientation) tracking with 2D touch input.

• Innovative Aspects of the Solution:

  1. HandyCast maps phone poses to the virtual environment, allowing users to precisely control virtual hand positions through small phone movements and touch gestures.
  2. Enables bimanual interaction even with a single smartphone.
  3. Supports space-constrained environments, offering efficient interaction while reducing physical space requirements.
  4. HandyCast is scenario-independent and compatible with existing VR applications without requiring additional hardware.

• Implementation Steps and Key Technologies:

  1. Pose Transfer:
     • Users define the phone's baseline position through a calibration point.
     • The vector of phone movement from the calibration point is nonlinearly amplified and mapped to the virtual hand's position.
     • The phone's orientation defines a 3D plane that serves as the basis for virtual hand operations.
  2. Touch Transfer:
     • Divides the touchscreen area, allowing the thumb of either hand to independently control the corresponding virtual hand.
     • Touch input is mapped with gain acceleration for precise control over longer distances.
  3. Touch Presence Transfer:
     • Touch events trigger virtual hand "grasp" and "release" commands.
  4. Uses SteamVR drivers to support integration with existing VR applications.

Research Outcomes

• Specific Achievements:

  1. Performance Evaluation:
     • HandyCast demonstrated efficiency in task completion time in both open space and space-constrained settings, performing as well as or better than classic dual-controller techniques (Go-Go).
     • In space-constrained scenarios, HandyCast completed tasks 22% faster than the improved Go-Go technique.
     • Physical movement paths were shorter, and the required physical control volume was significantly reduced.
  2. Summary of Experimental Results:
     • First User Study: HandyCast resulted in shorter paths and smaller control spaces compared to traditional dual controllers, with questionnaires indicating no compromise in virtual hand control or tactile experience.
     • Second User Study: In space-constrained settings, HandyCast significantly outperformed Space-Constrained Go-Go in terms of time and space efficiency.
     • Tracking Technology Study: HandyCast's inward tracking technology exhibited slight drift but remained practical.

• Advantages Over Existing Solutions:

  • HandyCast requires lower hardware demands and provides a more efficient interaction experience in small spaces, making it easily adaptable to mobile VR systems.

• Limitations and Future Directions:

  1. Inward tracking drift issues require dynamic calibration mechanisms to reduce cumulative errors.
  2. Current setup primarily supports simple bimanual tasks; future exploration could include more complex interaction tasks (e.g., more precise gesture control).
  3. Parametric impacts remain underexplored, and optimization of amplification and touch weight distribution parameters needs further study.
  4. Improvements to touch mapping methods could address issues with touch updates and rapid movements.