Mind the Kayak! Informing UX Design of Autonomous Vehicles through Edge Case Testing in the Field

Research Background and Issues

• What problems or challenges did the authors identify?

  1. With the deployment of autonomous vehicles (including waterborne transportation), ensuring user safety and trust has become a critical issue.
  2. Current research on user experience (UX) primarily focuses on virtual simulations, desktop studies, and surveys, lacking field tests in real-world scenarios.
  3. Autonomous ferries still face deficiencies in automation transparency and interaction design with other traffic participants (e.g., kayaks, boats).

• Why is this issue important?

  1. User trust and sense of safety in autonomous vehicles are fundamental to the acceptance of this technology.
  2. The development of autonomous ferries has potential positive impacts on urban transportation and environmental sustainability.
  3. Unresolved safety concerns and lack of trust may hinder the large-scale adoption of autonomous driving technologies.

• Research Motivation and Related Work

  1. Current virtual reality simulations and desktop studies provide low-cost, low-risk environments but fail to capture high-risk interactions and user feedback in real-world scenarios.
  2. Autonomous vessels in the maritime domain require further research on interactions in complex traffic environments, with kayaks considered typical high-risk traffic participants.
  3. This study aims to introduce "adversarial scenarios" through field tests to further explore user safety perceptions and trust in real operational environments.

Solutions

• What methods or solutions did the authors propose?

  1. Designed and implemented an adversarial scenario test simulating a collision scenario between a kayak and an autonomous ferry to evaluate user safety perceptions and trust.
  2. Combined quantitative surveys with qualitative interviews to refine the understanding of user experience.

• What is innovative about this solution?

  1. It is the first application of adversarial scenario testing in a real operational environment, addressing the gap in understanding user interactions in the field.
  2. Focused on passenger feedback regarding safety and transparency of autonomous ferries in risky scenarios, advancing research on external human-machine interfaces (eHMIs) in the maritime automation domain.

• What are the implementation steps and key technologies used?

  1. The experimental platform was the "milliAmpere2" autonomous ferry, equipped with autonomous navigation, dynamic positioning, and collision avoidance capabilities.
  2. Conducted a three-week public trial in the Trondheim fjord in Norway, including 20 kayak interference scenarios over two days.
  3. Observed passenger reactions to real high-risk traffic scenarios without prior knowledge.
  4. Data collection methods included post-experiment surveys (N=217, with experimental group N=39 and control group N=178) and semi-structured interviews (N=17).

Research Outcomes

• What specific outcomes were achieved?

  1. Passengers exposed to adversarial scenarios rated their sense of safety significantly higher when a safety operator was present compared to those who did not experience interference.
  2. Changes in safety perceptions were not significant in the absence of a safety operator, indicating that the performance of the autonomous ferry can enhance trust, but the presence of a safety operator is still considered important.
  3. While interference scenarios enhanced the sense of safety, they did not significantly change overall trust in the ferry or the willingness to recommend it.

• What advantages does this solution have compared to existing ones?

  1. Explored user reactions in real-world environments that virtual simulations and desktop studies could not capture.
  2. Field tests not only captured passengers' first impressions but also revealed directions for design improvements in complex environments.

• What were the experimental or evaluation results?

  • Quantitative data:
    • The kayak interference group showed a significant increase in safety perception scores for ferries with a safety operator (p=0.015), but no significant difference for ferries without a safety operator (p=0.165).
    • Trust in the ferry (p=0.358) and willingness to recommend (p=0.256) showed no significant differences between the interference and control groups.
  • Qualitative data:
    • Users found the ferry's response to kayaks overly cautious and suggested simplifying the response process.
    • Passengers desired increased transparency, such as displaying the ferry's status and future intentions via screens or lights, and more intuitive communication with surrounding traffic.

• Limitations and Future Directions

  1. Limitations:
     • The design of adversarial scenarios was relatively simple, involving only kayak interference.
     • Did not include subjective feedback from other traffic participants (e.g., kayakers).
     • The presence of a safety operator during the experiment may have mitigated the psychological challenge for passengers.
  2. Future Directions:
     • Introduce more complex scenario tests, such as multi-vehicle interactions or system failures.
     • Investigate the impact of invisible safety operators on board or develop applications for remote control centers.
     • Expand the scope of research to include perspectives of other traffic participants (e.g., small boat users) on interactions with autonomous ferries.
     • Further develop external human-machine interface (eHMI) designs, particularly for maritime applications.