ShadowTouch: Enabling Free-Form Touch-Based Hand-to-Surface Interaction with Wrist-Mounted Illuminant by Shadow Projection

Title of the Paper

ShadowTouch: Enabling Free-Form Touch-Based Hand-to-Surface Interaction with Wrist-Mounted Illuminant by Shadow Projection

Paper Information

• Domain: Human-Computer Interaction, Computer Vision, Virtual Reality and Augmented Reality Touch Technologies
• Keywords: Touch detection, hand-to-surface interaction, computer vision, mixed reality, shadow projection, gesture recognition

Research Background and Problem

• Issues or Challenges:
  • Current gesture recognition and hand interaction technologies face limitations in spatial and temporal resolution for near-surface touch detection in virtual reality environments.
  • Commercial hand tracking devices (e.g., Microsoft Hololens 2 and Oculus Quest 2) struggle to accurately distinguish between finger touch and hover states, especially during fast and subtle touch actions.
  • Although various existing technologies, such as inertial sensors, vibration patterns, and optical sensing, attempt to address touch detection, they fall short in flexibly recognizing the touch state of each finger.
• Significance:
  • Physical surface touch is a crucial interaction modality in virtual and augmented reality environments, offering intuitive operations and tactile feedback similar to touchscreens.
  • Accurate recognition of finger touch states near surfaces can enhance the precision of virtual interactions, improve user experience, and expand interaction possibilities.
• Research Motivation and Related Work:
  • The authors aim to develop a novel optical-assisted method to address the limitations of existing near-surface touch detection technologies.
  • Related studies, such as those utilizing vibration signals to recognize touch events or employing image capture, have demonstrated the potential of visual enhancement technologies.

Solution

• Proposed Solution:
  • ShadowTouch: An innovative vision-based touch detection technology that utilizes shadow projection from a wrist-mounted forward-facing light source.
  • This technology amplifies subtle vertical finger movements and converts them into shadow features recognizable by computer vision algorithms.
• Innovations of the Solution:
  • Actively constructing high-quality shadow features to transform small finger movements near surfaces into optically recognizable patterns.
  • Achieves multi-finger touch state detection, supporting multi-touch, sliding, and finger angle information.
• Implementation Steps and Key Technologies:
  1. Hardware Design:
     • Wrist-mounted LED light source for projecting finger shadows.
     • A head-mounted camera captures shadow images for further processing and classification of finger touch states.
     • High-frame-rate (120FPS) cameras ensure dynamic capture of touch processes.
  2. Algorithm Pipeline:
     • Step 1: Hand keypoint detection and finger region localization using the Mediapipe hand detection algorithm.
     • Step 2: Touch state classification. A lightweight neural network based on MobileNet V3 is proposed to achieve high-precision recognition of finger touch states.
  3. Data Collection and Annotation:
     • Real-time acquisition of ground truth touch state data using the Sensel Morph touchpad.
     • A high-quality annotated dataset is provided to support cross-user testing.

Research Results

• Specific Results:
  • Performance Metrics:
    • In cross-user evaluations, the touch state detection algorithm achieved 99.1% recognition accuracy and an F-1 score of 96.8%.
    • Compared to existing technologies, ShadowTouch significantly improves the resolution and computational efficiency of touch state detection.
  • Application Development:
    • Four prototype applications were developed:
      1. Virtual desktop application (navigator) supporting mouse-like control.
      2. Image processing application supporting multi-touch rotation, zooming, and panning.
      3. Drawing application enabling simultaneous use of multiple fingers with different brushes.
      4. Text input application integrating multi-gesture cursor control and keyboard input.
  • User Evaluation:
    • In usability assessments, users found ShadowTouch superior in touch precision, ease of use, and overall willingness to use compared to threshold-based touch methods.
    • ShadowTouch provides an efficient touch experience closely aligned with physical surfaces, significantly reducing user frustration and operational burden.
• Limitations and Future Directions:
  • Limitations:
    • The effectiveness of optical shadows depends on surface material; highly reflective or dark surfaces may reduce performance.
    • Finger occlusion in certain scenarios can lead to decreased accuracy in touch state recognition.
  • Future Directions:
    • Enhance system robustness across different surfaces and ambient lighting conditions.
    • Extend to dual-hand interaction designs and optimize user experience with invisible light sources.
    • Further explore interaction effects on irregular surfaces, such as vertical surfaces and complex geometries.

Conclusion

The research on ShadowTouch demonstrates new opportunities and methods for hand touch detection. By actively constructing shadow features, this system significantly improves the accuracy of finger touch state detection and expands interaction possibilities in virtual and augmented reality environments, particularly for physical surface interactions. This study lays a theoretical and technical foundation for the future development of related technologies.