Effects of Semantic Segmentation Visualization on Trust, Situation Awareness, and Cognitive Load in Highly Automated Vehicles

Title of the Paper

Effects of Semantic Segmentation Visualization on Trust, Situation Awareness, and Cognitive Load in Highly Automated Vehicles

Paper Information

• Field of Study: Human-Computer Interaction, Autonomous Driving Technology
• Keywords: Autonomous Vehicles, Semantic Segmentation, Augmented Reality, Trust Calibration, Cognitive Load, Situation Awareness, User Experience

Research Background and Issues

• Identified Problems and Challenges:

  • Trust issues in autonomous vehicles: insufficient or excessive trust can impact technology usage and safety.
  • The detection capabilities of autonomous driving technology are difficult for external users to perceive.
  • Current driver assistance systems may lead to inadequate user responses in emergency situations.

• Importance of the Problem:

  • The development of autonomous driving technology can significantly transform transportation, including improving safety, enhancing travel convenience, and assisting special groups (e.g., the elderly and disabled).
  • User acceptance and trust in the technology are critical for successful adoption.

• Research Motivation and Related Work:

  • Visualizing semantic segmentation information can help users perceive the detection capabilities of autonomous vehicles, thereby calibrating their trust.
  • Augmented reality technology can display the vehicle's understanding of the environment, enabling users to have higher situation awareness during sudden takeover tasks.

Solution

• Proposed Methods or Solutions:

  • Development of two semantic segmentation visualization techniques: augmented reality (AR) and tablet-based displays.
  • Exploration of visualization effects for detecting dynamic objects and dynamic + static objects.

• Innovative Aspects:

  • Proposing the use of semantic segmentation visualization to directly convey the detection capabilities and uncertainties of autonomous vehicles, rather than emphasizing detection accuracy.
  • Encoding various dynamic and static objects with different colors and visualizing them in different views (e.g., windshield and side windows).
  • Designing experiments and specific measurement methods to study the impact of the technology on user trust, situation awareness, and cognitive load.

• Implementation Steps and Technical Details:

  1. Conducting preliminary experiments in a simulated environment using Unity, designing five conditions for dynamic and dynamic + static displays.
  2. Conducting further experiments using real-world complex traffic environment videos, applying the advanced semantic segmentation model Panoptic-Deeplab.
  3. Testing metrics include trust evaluation, cognitive load tests, situation awareness scores, and user assessment of system detection capabilities.

Research Outcomes

• Specific Findings:

  • AR visualization significantly enhanced users' situation awareness, performing best in scenarios with dynamic and static object visualization.
  • Two experiments revealed that visualization schemes effectively calibrated users' trust in vehicle detection capabilities without excessively increasing trust.

• Advantages Compared to Existing Solutions:

  • Provides more concrete and visualized information compared to abstract representations, helping users directly evaluate vehicle detection capabilities.
  • Augmented reality is considered the ultimate goal for conveying information in autonomous vehicles, reducing cognitive load and improving user-friendliness.

• Experimental or Evaluation Results:

  • In simulation-based research, the AR system reduced cognitive load, enhanced situation awareness, and improved user trust.
  • In real-world video-based research, although trust improvement was limited, users' evaluations of system capabilities with AR visualization were significantly higher than systems without semantic segmentation information.
  • Detection scores for dynamic objects improved significantly, while detection capabilities for static objects were nearly equivalent to those for dynamic objects.

• Limitations and Future Directions:

  • The experimental sample primarily consisted of younger participants; future studies should include a broader age range.
  • In real-world video experiments, external validity may be affected by perspective differences and technical limitations (e.g., varying camera positions).
  • Future research is recommended to explore intent recognition in more complex scenarios and methods for visualizing driving information shared between vehicles.