HaloTouch: Using IR Multi-Path Interference to Support Touch Interactions with General Surfaces

Research Background and Issues

Issues and Challenges

1. Limitations of Touch Technology:

   • Current technologies, such as traditional keyboards and mice, have limitations in interaction flexibility.
   • Achieving touch interaction on general surfaces often requires hardware modifications, increasing deployment complexity.

2. Deficiencies of Existing Solutions:

   • Depth camera-based solutions reduce surface modifications but suffer from low detection accuracy, high latency, and large minimum hover distances, which limit their interaction capabilities.
   • Existing systems excel in specific metrics such as touch accuracy, latency, or material adaptability, but none comprehensively meet all design requirements.

3. New Demands in Human-Computer Interaction:

   • Users increasingly demand natural, easily deployable touch systems, but existing solutions lack adaptability and flexibility.

Research Significance

• Enhancing Generalizability: Achieving high-precision touch on unmodified surfaces can greatly improve system deployability and application scenarios.
• Improving Interaction Experience: Enabling comprehensive touch capabilities, including touch location, pressure sensing, and hovering, will pave the way for the integration of digital and physical worlds.

Research Motivation and Related Work

• Many technologies (e.g., Electrick, TapLight, Tripad) have successfully optimized specific touch metrics but fail to address multiple metrics simultaneously.
• The innovation of HaloTouch lies in utilizing the multi-path interference effect of commercial Time-of-Flight (ToF) depth cameras. It achieves touch and pressure detection without surface modifications or additional hardware.

Solution

Methodology and Implementation

• Principle:

  • Leverages the multi-path interference phenomenon of ToF depth cameras ("Halo Effect"), where interference signals create additional "halos" when detecting object depth.
  • This phenomenon is used to detect the position and depth of a finger approaching or pressing a surface.

• Key Steps:

  1. Background Modeling and Signal Correction:
     • Background signals are removed from captured depth frames to isolate moving objects (e.g., fingers).
     • Machine learning models correct nonlinear errors caused by different finger positions and postures.
  2. Calibration Mechanism:
     • Users perform a 20-second personalized calibration, including multiple states such as hovering and high-pressure touch.
  3. Multi-Mode Interaction Support:
     • Provides fine-grained "hover distance detection" and "pressure detection."
  4. Finger Signal Capture and Recognition:
     • Combines depth + RGB streams mapping with the Google MediaPipe hand tracking framework.

• Hardware Configuration:

  • Utilizes a Microsoft Kinect Azure depth camera and a commercial projector as primary hardware.

Innovations

1. First Exploration and Utilization of the Halo Effect:

   • Extends this phenomenon to touch, pressure, and hover detection, beyond traditional depth measurement applications.

2. Comprehensive Software and Hardware Optimization:

   • Requires no surface modifications or wearable hardware, compatible with various surface materials (e.g., plastic, wood, leather).
   • Enables instant, seamless touch and hover interactions.

Research Outcomes

Experiments and Results

1. Touch Accuracy:

   • Achieved an average touch accuracy of 99.2% across five different materials.
   • The average spatial positioning error is 5.5 mm.
   • System touch latency is 150 ms, below the threshold typically noticeable by users.

2. User Experience:

   • In user testing (including virtual keyboard typing tasks), the system achieved an input speed of 26.3 AWPM (Adjusted Words Per Minute), comparable to devices with more hardware dependencies.

3. Pressure and Hover Detection:

   • The average hover distance error is only 2.81 mm, and the pressure detection error is 18.77%.
   • The system can distinguish between different pressure levels and hover heights.

4. Additional Innovation Testing:

   • Applications such as typing and drawing demonstrated the system's ability to support dynamic keyboard interactions and pressure-sensitive drawing in multi-mode interactions.

Comparative Advantages Over Existing Solutions

• Compared to other systems like Electrick and TapLight, HaloTouch features a lower touch point threshold (4.97 mm), enabling faster and more accurate touch input.
• Strong material compatibility without requiring additional hardware or sensors.

Limitations and Future Directions

• User Privacy Concerns:

  • Further exploration is needed to protect user privacy (e.g., typing content privacy).
  • Migrating the system to AR/VR devices may be a feasible solution.

• Environmental Reliability:

  • More advanced signal processing and calibration models may be required for complex environments (e.g., large areas, multi-user scenarios).

• Deployment Improvements:

  • Current manual calibration could be replaced with more automated or calibration-free models in the future.
  • Expand support to a broader range of hardware configurations, such as mobile devices.

Conclusion

HaloTouch successfully achieves multi-mode touch interaction on general surfaces by uniquely leveraging the multi-path interference phenomenon of commercial depth cameras. Its performance indicates potential integration into AR/VR, smart homes, and other fields. However, further improvements are needed in privacy protection, adaptability to complex scenarios, and user learning curves for broader application and optimization.