PatternTrack: Multi-Device Tracking Using Infrared, Structured-Light Projections from Built-in LiDAR

Research Background and Problem Statement

• Identified Issues or Challenges:
  Currently, achieving multi-user shared and synchronized interactive experiences in augmented reality (AR) is highly challenging. It often requires complex equipment coordination and scene preprocessing, such as utilizing visual fiducial markers (physical markers), UWB or Bluetooth for proximity sensing, or SLAM (Simultaneous Localization and Mapping) technology, which involves cumbersome initialization scans. These methods frequently fail on featureless surfaces (e.g., smooth walls).

• Significance:
  As AR devices (e.g., smartphones, head-mounted displays) become increasingly popular, the demand for multi-user collaborative AR scenarios is growing. Addressing these challenges is crucial for enhancing AR user experiences and expanding application scenarios—from healthcare to gaming and industrial collaboration.

• Research Motivation and Related Work:
  The authors argue that existing methods are time-consuming, dependent on external devices or infrastructure (e.g., physical markers or local scanning), and difficult to apply to specific scenarios (e.g., smooth surfaces). This study aims to leverage built-in LiDAR sensors and infrared structured light projection in consumer devices (e.g., iPhone or Meta Quest 3) to address these issues, enabling plug-and-play multi-device tracking.

Solution

• Proposed Method or Solution:
  The authors introduce PatternTrack, a novel 6DOF (six degrees of freedom) multi-device positioning method based on the infrared projection patterns of built-in LiDAR sensors. By capturing the infrared dot projection patterns of nearby devices and analyzing their perspective distortions, PatternTrack can estimate the position and orientation of other devices in 3D space.

• Innovative Features:

  1. No Infrastructure or Physical Markers Required: Eliminates the need for printed markers, base stations, or other external devices.
  2. Adaptability to Featureless Surfaces: Operates effectively on "featureless" areas such as smooth or white surfaces.
  3. Quick Startup: Achieves spatial positioning with single-frame data, avoiding time-consuming scene scanning or pre-registration processes.
  4. Optimized for Short-Range Collaboration: Provides real-time 6DOF tracking within a range of 0.5–2.5 meters, with accuracy comparable to device dimensions.

• Implementation Steps and Key Technologies:

  1. Utilize the dot pattern projected by LiDAR structured light, capturing these dots with an infrared camera.
  2. Generate 3D point clouds using depth cameras and extract projection patterns from multiple devices.
  3. Solve the Perspective-n-Point problem (PnP algorithm) using single-frame data to compute the 6DOF movement and rotation of devices.
  4. Separate overlapping dot patterns from multiple devices using time multiplexing or deep learning models.

Research Outcomes

• Specific Results Achieved:

  1. Tests show that PatternTrack achieves an average 3D position tracking error of 11.02 cm and an angular error of 6.81° across six common surfaces (e.g., white walls, blue walls, gray tables).
  2. Spatial positioning is achieved using single-frame data, offering faster startup compared to methods like SLAM that require preprocessing.
  3. Successfully deployed a proof-of-concept prototype, including a modified iPhone, infrared cameras, and supporting algorithms.

• Advantages Compared to Existing Solutions:

  1. Unlike SLAM and cross-device environment scanning, PatternTrack requires only a single frame, eliminating the need for time-consuming environment mapping processes.
  2. Compared to UWB and Bluetooth methods, PatternTrack provides true 6DOF tracking rather than mere distance measurements between devices.
  3. Does not require additional physical markers or reliance on external hardware infrastructure such as depth cameras or base stations.

• Experimental or Evaluation Results:

  1. Surface Impact: Performs well on surfaces like gray tables and wooden tables but shows reduced performance on materials with strong infrared absorption (e.g., dark concrete).
  2. Distance Impact: Accuracy decreases slightly when the viewing device and projection surface distance reaches 1.5 meters, but remains within acceptable limits.
  3. Angle Impact: Accuracy diminishes at shallow viewing angles (e.g., 30°) due to pattern distortion, but still provides reasonable 6DOF estimates.

• Limitations and Future Directions:

  1. Surface Compatibility: The current method is unsuitable for transparent surfaces (e.g., glass) or very dark materials. Future improvements could involve optimizing camera parameters or using higher-performance sensors.
  2. Multi-Device Management: Enhancements are needed in managing LiDAR dot pattern timing to avoid interference from overlapping patterns between devices.
  3. Computational Requirements: The current algorithm achieves only 8 FPS on modern laptops, necessitating optimization for real-time performance.
  4. Expanded Application Domains: Future research could integrate SLAM, UWB, and other technologies to enable tracking over longer distances or in more complex collaborative scenarios.

Conclusion

PatternTrack leverages existing LiDAR projection patterns in consumer devices to enable instant multi-device 6DOF tracking, significantly reducing the startup complexity and hardware requirements for multi-user AR collaboration. Experimental results demonstrate promising accuracy, particularly in typical short-range collaboration environments (less than 2.5 meters). Future optimization efforts will focus on improving algorithm efficiency, supporting more material types and complex scenarios, and enhancing device collaboration capabilities. This approach has the potential to become a universal, barrier-free solution for multi-user AR interaction.