ReLive: Bridging In-Situ and Ex-Situ Visual Analytics for Analyzing Mixed Reality User Studies

Title of the Paper

ReLive: Bridging In-Situ and Ex-Situ Visual Analytics for Analyzing Mixed Reality User Studies

Paper Information

• Domain: Human-Computer Interaction, Mixed Reality, Visual Analytics
• Keywords: Mixed Reality Analytics, Immersive Analytics, Visual Analytics, Data Visualization, Virtual Reality, User Research Tools, Contextual Analysis, Linked Interaction, Open Data Logging

Research Background and Problem Statement

• Problems or Challenges:

  1. The rapid development of the Mixed Reality (MR) field requires analyzing complex and intersecting datasets such as user behavior, spatial usage, and motion patterns, while also incorporating traditional HCI performance metrics (e.g., time and error rates).
  2. MR research is characterized by device diversity, strong environmental context dependency, and complex data forms. Existing analytical frameworks typically support either immersive or non-immersive analysis, lacking transitional tools for cross-environment analysis.
  3. MR research often necessitates constructing the context of user behavior within experimental environments, yet current tools lack support for transitioning from immersive analysis to data aggregation analysis.

• Significance: MR research imposes new demands on experimental research infrastructure in fields such as computer science, human-computer interaction, and data analysis. Cross-environment analytical tools can enable in-depth exploration of data environments and promote open science and research transparency.

• Motivation and Related Work: The authors integrate the strengths of immersive tools (e.g., augmented reality and virtual reality devices) and traditional desktop analysis tools to propose a hybrid framework addressing the aforementioned issues. Related work includes:

  1. Using immersive analytics (3D trajectories, event visualization) to study user behavior and environments;
  2. Non-immersive analysis tools (e.g., Tableau) for data grouping and visualization;
  3. A lack of research enabling rapid transitions between mixed reality environments.

Proposed Solution

• Method: The authors propose a hybrid immersive visual analytics framework called ReLive, which integrates tools from immersive environments (virtual reality) and non-immersive environments (desktop) to support comprehensive exploration and flexible analysis of user study data.

• Innovations:

  1. Cross-environment synchronization: Real-time synchronization of states between virtual reality and desktop environments, enabling seamless transitions.
  2. Data exploration: Combines environmental re-enactment in virtual reality with comprehensive overviews in desktop analysis, providing bidirectional perspectives.
  3. Extensibility: Modular design allows for customizable analysis workflows, supports open data standards, and facilitates interactive visualizations.
  4. Data logging tools: Supports unified formatting and conversion of experimental data, enhancing compatibility and reusability across tools.

• Key Techniques and Implementation Steps:

  1. Data Standardization: Defines standardized data structures (e.g., sessions, entities, events) to facilitate analysis after experimental data collection.
  2. Interface Transition and Interaction Support: Enables seamless migration between environments through components such as timelines, editable trajectory tools, and analysis templates; user interfaces include desktop, video playback, and 3D scenes.
  3. Prototype Development and Real-Time Synchronization: Establishes a client/server architecture where desktop and VR function as clients, synchronized in real-time via protocols such as WebSocket and HTTP.

Research Outcomes

• Achievements: The authors validated the concept and practical application of ReLive through a two-phase evaluation:

  1. Design walkthrough: Conducted functionality prioritization and prototype testing based on five reference MR experiments.
  2. Expert user study: Five MR domain experts participated, providing feedback on environment transitions, user experience, and task allocation, including migration costs and efficiency in problem-solving.

• Strengths and Weaknesses:

  1. Strengths: The ReLive framework enables flexible data analysis, supports contextual reasoning and exploration, and its modular design allows researchers to extend analytical content. It enhances the transparency and reproducibility of MR experiment analysis.
  2. Weaknesses: Certain tools require further optimization, such as component allocation methods and consistency in terminology. Using VR headsets may cause discomfort for some users. Further research is needed on programming extensions for components.

• Experimental Results: Experts provided an average system usability score (SUS) of 74.5. Qualitative data analysis revealed that VR performed better for contextual reasoning, while desktops were more suitable for global navigation and comparative tasks.

• Limitations and Future Directions:

  1. Investigate how "linked and interactive brushing" in cross-reality environments can improve data analysis efficiency.
  2. Further study user behavior during transitions between desktop and VR, particularly focusing on the continuity of the switching experience.
  3. Expand compatibility with open science, exploring mechanisms for sharing component templates and further refining open data standards.