UI Mobility Control in XR: Switching UI Positionings between Static, Dynamic, and Self Entities

Document Title

UI Mobility Control in XR: Switching UI Positionings between Static, Dynamic, and Self Entities

Document Information

• Subject Area: User interface mobility and interaction design in extended reality (XR)
• Keywords: UI mobility, extended reality, virtual reality, user interface interaction, gesture control, mode switching, embodied interaction

Research Background and Issues

• Issues and Challenges: Current research on the mobility of 3D user interfaces (UI) in extended reality is insufficient, lacking exploration of the design space, application scenarios, and user perception for seamless transitions of UI across people, objects, and environments. Moreover, commercial XR applications rarely support users in freely switching UI across entities.
• Importance: As XR technology advances, UI is no longer confined to physical screens or devices, opening up new possibilities and potential challenges for user interaction and UI design. Exploring how to optimize this 3D UI mobility can enhance user experience and meet the needs of diverse scenarios.
• Research Motivation and Related Work: The authors reviewed design frameworks for spatial UI positioning, research on UI mobility, and studies on mode switching techniques in XR. They identified a research gap in achieving comprehensive UI mobility control across "static, dynamic, and self" entities.

Solution

• Proposed Method: The authors defined the concept of "3D UI mobility," describing the switching of UI between static entities, dynamic entities, and self entities (three host entity types). They developed an interaction technique called FingerSwitches, which combines user gaze and gesture operations to enable users to specify the target entity for the UI in 3D space.
• Innovations:
  • Proposed a comprehensive classification of 3D UI mobility, covering nine mobility modes between static, dynamic, and self entities.
  • Developed the FingerSwitches technique, which uses finger position and gaze location to predict the target entity type for mode switching.
  • Designed practical prototypes and applied them to specific scenarios such as classrooms, meeting rooms, and national parks.
• Key Technologies and Steps:
  1. Defined types of UI host entities and categorized application cases into UI mobility across static, dynamic, and self entities.
  2. Conducted a survey of 113 XR applications to identify shortcomings in current UI mobility control.
  3. Implemented the FingerSwitches technique, allowing users to select UI via gaze, adjust the target entity type using gestures, and complete interactions.
  4. Verified prototypes and conducted user studies using Meta Quest Pro.

Research Outcomes

• Main Findings:
  • Identified the design space, key scenarios, and application needs for 3D UI mobility in XR, summarizing a classification framework centered on user host entities.
  • Survey results from commercial XR applications revealed that only 6.2% of applications support UI mobility across entities, with none fully covering all nine possibilities.
  • User evaluations confirmed the effectiveness and usability of the FingerSwitches technique across various scenarios.
• Advantages:
  • Compared to existing solutions, the FingerSwitches technique offers more comprehensive 3D UI mobility control, enabling seamless UI switching across static, dynamic, and self entities.
  • The system provides smooth transitions and precise control, particularly suitable for professional or entertainment purposes in dynamic scenarios.
• Experiment/Evaluation Results:
  • User studies showed strong demand for modes such as static-to-self and dynamic-to-static, with high recognition of dynamic-to-dynamic sharing behaviors.
  • Applicability evaluations varied significantly across scenarios, with formal settings like conferences being more favored.
• Limitations and Future Directions:
  • Limitations: Current XR device hardware imposes constraints on computational capabilities, leading to some inaccuracies in gesture tracking; participant backgrounds were not diverse enough, leaving long-tail applications insufficiently validated.
  • Future Directions:
    1. Improve hand tracking and UI occlusion issues to enhance technical robustness.
    2. Strengthen privacy and data protection mechanisms to address potential security risks during UI sharing.
    3. Conduct in-depth optimization and extended design for specific industries and scenarios, such as medical training and immersive data analysis.