Structured Light Speckle: Joint egocentric depth estimation and low-latency contact detection via remote vibrometry

Title of the Paper

Structured Light Speckle: Joint egocentric depth estimation and low-latency contact detection via remote vibrometry

Paper Information

• Subject Area: Haptic interaction and depth detection technologies in Virtual Reality (VR) and Mixed Reality (MR)
• Keywords: Virtual Reality, Mixed Reality, Augmented Reality, Haptic sensing, Laser vibrometry, Surface reconstruction, Structured light

Research Background and Problem Statement

• Identified Problems and Challenges:

  1. Current hand-tracking technologies in head-mounted devices struggle to accurately detect rapid touch events, especially when users interact with physical surfaces that lack sensing equipment.
  2. Existing solutions often require additional sensors or complex system integration, increasing device cost and complexity.
  3. Detecting surface touch from a centralized perspective in dynamic and mobile scenarios is highly challenging.

• Significance: Leveraging the sensing capabilities of existing VR devices to directly detect physical contact can enhance user immersion and interaction efficiency in virtual environments, reduce user fatigue, and improve the fluidity of the interaction experience.

• Research Motivation: Utilizing optical technologies (e.g., structured light and laser speckle) to achieve touch detection, thereby reducing dependency on additional hardware while improving the accuracy and latency performance of touch input.

• Related Work:

  1. Research on gesture tracking and physical surface interaction (e.g., using depth cameras, infrared technologies).
  2. Detection of surface vibrations and physical contact using laser speckle and remote vibrometry techniques.
  3. Exploration of issues in current depth-based sensing technologies, such as high noise levels, low accuracy, and significant latency.

Proposed Solution

• Methodology and Innovations:

  1. A novel method called "Structured Light Speckle" is proposed, combining depth estimation from laser-structured light and vibration detection from laser speckle dynamics.
  2. Speckle imaging technology is utilized for remote vibrometry, enabling reliable detection of touch events under dynamic conditions.
  3. An integrated system, "TapLight," is designed to combine these technologies into a portable device that can be mounted on head-mounted displays.

• Implementation Steps and Key Technologies:

  1. Depth Estimation:

     • Employ structured laser light sources (e.g., diffraction grating-generated dot grids) to generate depth estimations.
     • Extract laser reflection points and calculate sparse depth maps using the parallax between the camera and laser emitter.
     • Fit depth points to create planar models of physical surfaces.

  2. Contact Detection:

     • Capture dynamic changes in laser speckle patterns using cameras to extract surface vibration signals.
     • Detect changes in speckle pattern roughness signals to identify touch events.
     • Combine hand position and velocity data from the head-mounted display to validate touch events and determine touch locations.

  3. System Integration:

     • Integrate the sensing modules with VR rendering environments to provide low-latency touch input through real-time data interaction.

Research Outcomes

• Specific Results:

  1. TapLight achieves surface discovery and touch detection in dynamic environments, with an overall F1 score of 0.953 for touch events and an average latency of 50.4 milliseconds.
  2. The system's depth error for planar detection within a 1-meter range is 22 mm (horizontal surfaces) to 45 mm (vertical surfaces), with a surface normal angle error of 2.8° (horizontal surfaces).

• Comparison with Existing Solutions:

  1. Compared to traditional depth cameras and inertial sensor-based solutions, TapLight offers lower latency (50.4 ms), outperforming systems like Hololens or Dante multimodal sensing systems (typically >200 ms).
  2. The system eliminates the need for additional wearable hardware, simplifying device configuration while maintaining operational flexibility in dynamic conditions.

• Experimental or Evaluation Results:

  1. In depth estimation experiments, the system successfully reconstructed surfaces with high accuracy across different materials.
  2. In touch detection experiments, the system demonstrated stable performance across various surface materials (e.g., wood, plastic, cardboard).
  3. Signal attenuation within approximately 1 meter was minimal, making the system suitable for scenarios within user interaction distances.

• Limitations and Future Directions:

  1. Limitations:

     • Currently uses visible laser light, which may pose limitations in multi-user scenarios; infrared light sources are recommended for future iterations.
     • The resolution of the grating limits depth estimation to sparse data, potentially creating bottlenecks in detecting complex surfaces.
     • The system is currently limited to detecting index finger touches and has not been extended to multi-finger or complex gesture detection.
     • Most application scenarios involve seated users, with limited validation for standing or walking conditions.

  2. Future Directions:

     • Explore higher-resolution laser optical systems to improve depth perception accuracy and density.
     • Expand the system to broader dynamic scenarios, such as dynamic interactions or outdoor environments.
     • Investigate methods to capture complex haptic features such as touch force and shape.
     • Integrate machine learning algorithms to support richer input modes and event classification.