AutoVis: Enabling Mixed-Immersive Analysis of Automotive User Interface Interaction Studies

Title of the Paper

AutoVis: Enabling Mixed-Immersive Analysis of Automotive User Interface Interaction Studies

Paper Information

• Subject Area: Interaction Analysis and Automotive User Interface Research
• Keywords: Immersive Analysis, Interaction Analysis, Visualization, Virtual Reality, Automotive User Interface

Research Background and Problem

• Research Background: The evaluation of automotive user interfaces (AUI) has become increasingly complex, involving multiple interaction modes, driving automation, heterogeneous data, and dynamic environmental contexts. Current analysis tools are often non-immersive and fail to intuitively capture the complexity of human-vehicle-environment interactions.
• Identified Challenges:
  • Existing tools are typically desktop-based and non-immersive, lacking the ability to correlate spatial systems and contextual associations.
  • Inefficient handling of spatiotemporal data, with limited integration of observer attention, behavior, and environmental information.
  • The heterogeneity of data makes the processing and analysis of public datasets lengthy and complex.
• Research Motivation: To provide a method that combines immersive analysis (via VR environments) with non-immersive analysis (desktop views) to address the shortcomings in data visualization and analysis commonly found in AUI research.

Solution

• Method and Approach:
  • A new tool named AutoVis is proposed, combining virtual reality and desktop views for analyzing automotive user interface interaction studies.
  • AutoVis supports the analysis of passenger behavior, physiological signals, spatial interactions, and experimental events, integrating deep learning-based data preprocessing capabilities.
  • Features include virtual vehicle and environment visualization, role simulation, trajectory animation, heatmap displays, and enhanced associations between data and environment.
• Innovations:
  • Integration of non-immersive and immersive analysis, enabling seamless switching between VR analysis and desktop tools.
  • Development of specific 3D visualization concepts tailored for the automotive domain, such as Context Portals and Driving-path Events.
  • Support for real vehicle-based analysis, enhancing realism and tactile interaction through "Passthrough VR" technology.
• Key Technologies and Implementation Steps:
  • Data Preprocessing: Using deep learning models (e.g., OpenPose, YOLOv4, DeepFace) to automatically identify events, emotions, objects, and actions. Virtual environments are constructed using data from sources like OpenStreetMap.
  • Non-immersive Desktop View: Provides an overview of data, supporting event filtering, timeline control, 2D charts, and 3D scene reconstruction of experimental environments.
  • Immersive VR View: Enables detailed observation of experimental reconstructions, including role simulation, 3D trajectories, heatmaps, and Context Portals.
  • Cross-device and Multi-user Collaboration: The system allows seamless switching between desktop and VR views for both individual and collaborative analysis.

Research Outcomes

• Specific Results:
  • The AutoVis prototype demonstrated applicability across multiple scenarios for real experimental tasks.
  • Successfully achieved multimodal interaction visualization analysis (e.g., gestures, voice, gaze, and environmental interactions).
  • Showcased the ability to convert and analyze real-world datasets (e.g., Drive&Act).
  • Enabled data analysis in real vehicle scenarios, combining VR Passthrough to enhance spatial and object interaction experiences.
• Advantages:
  • Compared to existing tools, AutoVis significantly improves the quality of data visualization and analysis efficiency.
  • The innovative mixed-immersive framework addresses the limitations of traditional tools in associating data with environmental context.
• Experiments and Evaluation:
  • Validated the feasibility of the concept and the usability of the tool through real-world user case studies.
  • Using Olsen's heuristic evaluation, the tool demonstrated advantages in task allocation, collaboration support, and multimodal interaction visualization.
  • Successfully handled various use cases, including multimodal autonomous driving interactions, real-world dataset analysis, and studies conducted using real vehicles.
• Limitations and Future Directions:
  • Data quality and completeness are critical challenges in generating visualizations, as some datasets may lack environmental recordings or detailed spatiotemporal data.
  • Current VR device resolutions may not fully meet practical analysis requirements and need further optimization.
  • To handle larger and more complex datasets, future work could include intelligent filters, partitioned analysis methods, and adaptive views.
  • Further validation is needed to explore the tool's applicability in other automotive interaction research areas, such as external HMI and driver distraction scenarios.