STREAM: Exploring the Combination of Spatially-Aware Tablets with Augmented Reality Head-Mounted Displays for Immersive Analytics

Document Title

STREAM: Exploring the Combination of Spatially-Aware Tablets with Augmented Reality Head-Mounted Displays for Immersive Analytics

Document Information

• Subject Area: Augmented reality technology and interactive visualization for data analysis (immersive analytics)
• Keywords: augmented reality, immersive analytics, multimodal interaction, mobile devices, data visualization

Research Background and Issues

• Key Issues or Challenges Identified
  • Augmented reality head-mounted displays (AR HMDs) are widely used for immersive data analytics, but their default mid-air gesture interactions suffer from a lack of precision, fatigue, and unreliability.
  • Traditional mobile devices offer higher precision with touch interaction but are limited to two-dimensional (2D) input, lacking support for three-dimensional (3D) interaction.
• Why It Matters
  • Addressing these issues can enhance the interaction experience for 3D data visualization, enabling users to perform more efficient and intuitive complex data analysis.
  • Combining spatially-aware tablets (tablets with spatial position sensing capabilities) with AR HMDs could open new possibilities for multimodal interaction and improve user experience.
• Research Motivation and Related Work
  • Existing studies mainly focus on hybrid user interfaces and the combined use of multiple devices, but most lack seamless interaction design between AR and touch devices.
  • Multimodal technologies (e.g., touch, voice) have shown significant advantages in extending interaction methods for information visualization, necessitating further exploration of their applications in AR environments.

Solution

• Method or Solution
  • Proposed the STREAM (Spatially-aware Tablets combined with Augmented Reality Head-Mounted Displays) system, which combines AR HMDs and spatially-aware tablets to support multimodal interaction with 3D data visualization.
  • Developed a new "eyes-free interaction" concept, allowing users to avoid frequent switching between different output interfaces.
• Innovations
  • Provided seamless interaction transitions between AR environments and tablet interfaces.
  • Combined the touch input of spatially-aware tablets with AR HMDs' gaze navigation and 3D layout operations.
  • Introduced spatial triggers and dynamic view-switching features (e.g., "tablet lens") based on direct tablet interaction.
• Implementation Steps and Key Technologies
  • Used 3D parallel coordinates to visualize multidimensional data, combined with 2D scatterplot input and output.
  • Offered interaction methods such as touch menus, voice input, and viewpoint selection.
  • Enabled object selection via gaze tracking on the head-mounted device, with further operations controlled through tablet touch input.

Research Outcomes

• Specific Results
  • STREAM successfully implemented a multimodal interaction scheme, receiving positive feedback on its usability and flexibility.
  • The system supports familiar 2D interactions (e.g., scatterplot manipulation) while leveraging the spatial awareness of tablets for 3D interactions.
  • The introduced eyes-free interaction system significantly reduced the switching cost between touchscreens and tablets.
• Comparison with Existing Solutions
  • Compared to traditional AR mid-air gestures, STREAM improved input efficiency and interaction precision through tablet touch.
  • Delivered a more flexible and mobile interaction experience than existing tablets or smartphones.
• Experimental or Evaluation Results
  • Quantitative and qualitative data from user studies indicated that most users found STREAM to provide an intuitive operational experience, despite requiring a certain learning curve.
  • Users generally appreciated the 2D scatterplot interactions and 3D object layout arrangement functionality.
  • A small number of users identified the physical weight of the tablet as a potential limiting factor.
• Limitations and Future Directions
  • Hardware limitations, such as the relatively heavy tablet device and the limited AR field of view, caused fatigue and restricted usage duration.
  • Future work should further investigate the accuracy and user comfort of different spatial interaction methods.
  • More granular collaborative scenarios (e.g., asynchronous multi-user or shared screen) require deeper exploration.