Phonetroller: Visual Representations of Fingers for Precise Touch Input when using a Phone in VR

Document Title

Phonetroller: Visual Representations of Fingers for Precise Touch Input with Mobile Phones in VR

Document Information

• Topic Area: Human-computer interaction design and touch input technology in virtual reality
• Keywords: Virtual Reality (VR), mobile phones, touch input, visual feedback, deep learning

Research Background and Problem

• Identified Problem or Challenge: Using mobile phones as touch input controllers in fully immersive VR environments suffers from precision issues because users cannot directly see the phone screen or their fingers, making it difficult to accurately target objects.
• Importance: Current VR controllers and mobile phone touch interfaces fail to provide precise touch operations, which poses significant limitations for tasks requiring high accuracy (e.g., keyboard input, selecting small tool buttons). This represents a critical technological bottleneck for enhancing the VR user experience.
• Research Motivation and Related Work:
  • Previous studies have demonstrated that visual feedback is crucial for improving the efficiency and accuracy of touch input.
  • Certain experimental devices (e.g., Leap Motion) support 3D hand tracking, but these devices are cumbersome and unsuitable for mobile environments.
  • The potential for optimizing basic mobile device functionalities (e.g., front cameras and simple hardware designs) has not been thoroughly explored.

Solution

• Proposed Solution: The authors propose a low-cost system, "Phonetroller," which captures finger dynamics before touch by installing a mirror structure above the phone to reflect the user's palm and thumb, and uses deep learning to infer hover points.
  • A mirror is mounted above the phone screen, and the phone's front camera captures a top-down view of the user's hand and thumb.
  • Two visual feedback techniques are provided: semi-transparent thumb shadow display and hover point inference using deep learning.
  • Supports three interaction scenarios: tool selection and 3D modeling, drawing and interaction, and gaming operations in VR mode.
• Innovative Aspects:
  • A simple hardware design combining mirror reflection and the phone's front camera, offering lower cost and greater ease of use compared to existing complex sensor devices.
  • Deep learning-based hover point inference enhances touch event precision and interactive experience.
  • Real-time finger visual feedback based on mirror-reflected images enables precise touch on small targets.
• Implementation Steps:
  • Use the front camera to capture images of the palm and fingers via mirror reflection.
  • Apply chromakeying background segmentation to extract the user's thumb image.
  • Implement real-time semi-transparent thumb shadow display and combine it with deep learning to infer hover points for operational feedback.
  • Demonstrate various VR application scenarios, such as data annotation, gaming operations, and pen interaction.

Research Outcomes

• Specific Results:
  • User experiments demonstrated that the Phonetroller system significantly improves touch accuracy, especially for small thumb-sized targets.
  • The deep learning method achieved an average hover point positioning error of 1.72 mm.
  • User evaluations confirmed the practicality and user experience of Phonetroller applications (e.g., Sketch and Paste, virtual pen interaction).
• Advantages Over Existing Solutions:
  • More accurate and stable feedback compared to traditional hardware-based hover detection devices (e.g., Samsung Galaxy S5's Air View feature).
  • Low cost and easy integration into existing devices.
  • Visual feedback significantly reduces the time required for users to touch targets while minimizing accidental touches.
• Experimental or Evaluation Results:
  • Touch methods without visual feedback (hud-blind) took more time, whereas systems with thumb shadow feedback (phone-shadow, hud-shadow) showed faster completion times.
  • Users highly appreciated the interaction design for applications like 3D modeling and gaming functions but noted that device weight and center of gravity affected long-term usage comfort.
• Limitations and Future Directions:
  • Limitations:
    • No 3D finger model is provided, and touch feedback remains constrained by the lack of depth information.
    • The deep learning approach performs best with more advanced hardware.
    • The mirror reflection design currently compromises device portability.
  • Future Directions:
    • Develop hardware based on convex mirror structures to improve viewing angles and device balance.
    • Further explore the potential of 6DoF mobile sensors and software tracking to reduce reliance on external trackers.
    • Investigate hybrid control modes (e.g., combining pen interaction and touch) and more application scenarios (e.g., deeper integration of mobile apps into VR environments).