Ubi Edge: Authoring Edge-Based Opportunistic Tangible User Interfaces in Augmented Reality

Document Title

Ubi Edge: Authoring Edge-Based Opportunistic Tangible User Interfaces in Augmented Reality

Document Information

• Topic Area: Design and implementation of Tangible User Interfaces (TUI) in augmented reality technology
• Keywords: Tangible User Interface, Augmented Reality, end-to-end programming, interaction design, visual programming, image and object detection, user research, system evaluation, Internet of Things (IoT), edge detection

Research Background and Problems

• Identified Problems or Challenges:

  • Current applications of physical object-based Tangible User Interfaces (TUI) are limited to strict matching between physical structures and digital functionalities.
  • Predefined TUIs lack generality and scalability, making it difficult to meet diverse user needs.
  • Some detection and interaction methods have intrusive effects on the functionality of physical objects or require object marking.
  • Traditional TUI inputs often struggle to seamlessly integrate with the local geometric features of everyday objects, particularly in edge detection and interaction control.

• Importance:

  • Edges are widely present in everyday objects, offering clear tactile feedback and efficient computer vision characteristics, enhancing the flexibility of TUI design.
  • Exploring the use of edges as tangible interaction interfaces can improve the practicality of augmented reality (AR) in scenarios such as smart homes, gaming, and education.

• Research Motivation and Related Work:

  • This study introduces "Ubi Edge," an innovative AR system designed to customize edges as interactive inputs for digital services, achieving high usability through markerless, efficient edge detection and interaction tracking.
  • It expands current TUI research by incorporating interaction methods based on local geometric features: utilizing edges as lines, sliders, or buttons to provide diverse user controls.

Solution

• Methods or Solutions:

  • Designed an integrated AR head-mounted display (AR-HMD) combining LiDAR and RGB-D vision detection technologies.
  • Developed an integrated edge detection and tracking pipeline, including high-precision RGB-D edge detection algorithms, lightweight neural networks for foreground-background separation, 6DoF object pose estimation, and edge-based iterative closest point (ICP) matching.
  • Introduced a touch-demonstration-based "program-action trigger" model, enabling users to flexibly select and customize edge interactions.
  • Provided an immersive AR user interface allowing users to intuitively create, edit, and adjust edge interaction functionalities.

• Innovations:

  • Proposed an end-to-end authoring tool enabling users to customize TUI inputs by sliding or tapping the edges of physical objects.
  • Integrated AR-HMD supports markerless real-time edge detection and interaction tracking, avoiding intrusive existing methods.
  • Utilized edges as sliders, buttons, and continuous interaction control inputs, significantly improving TUI scalability.

• Implementation Steps and Key Technologies:

  1. Edge Registration: Capturing RGB-D images of objects using LiDAR cameras and detecting geometric edges.
  2. Edge Matching: Real-time tracking of the object's 6DoF pose and performing ICP matching to ensure edge position accuracy.
  3. Input-Output Model: Defining discrete or continuous input edges and mapping them to digital behaviors through touch.
  4. TUI Creation: Users connect edges to preset digital behaviors in the AR interface, completing customization via visual programming.

Research Outcomes

• Specific Outcomes:

  • Successfully developed a complete system capable of detecting object edges and supporting users in customizing augmented reality interaction interfaces through touch.
  • Provided an efficient edge interaction mode, including basic operations such as sliding and clicking.
  • Demonstrated the potential of "Ubi Edge" in various application scenarios, such as smart home control, gaming, tutorials, etc.

• Advantages Compared to Existing Solutions:

  • Avoided intrusive object marking, supporting markerless real-time detection and interaction.
  • Integrated smooth edge selection and user-friendly touch definition methods, making TUI design more intuitive.
  • Offered a wide range of behavior combination options, supporting complex digital content control.

• Experimental or Evaluation Results:

  • Achieved superior performance in edge detection accuracy tests, with high F1 scores (e.g., F1 score = 0.88 for 45° angled edges).
  • User research data showed that 93.57% of touch operations were successfully detected; the average time to complete target tasks using continuous edges was 17.59 seconds.
  • User evaluation of system usability (SUS) scored 87.86/100, indicating broad user acceptance.

• Limitations and Future Directions:

  • The system's edge detection capability is weaker for transparent materials or symmetrical objects.
  • In complex environments, edges may be occluded or out of the field of view, leading to detection failures.
  • Improvements are needed in mobility and lightweight device design.
  • Authoring behavior logic requires further enhancement, such as supporting conditional triggers and sequential connections.