eHMI for All - Investigating the Effect of External Communication of Automated Vehicles on Pedestrians, Manual Drivers, and CyclistsWith automated vehicles (AVs), the absence of a human operator could necessitate external Human-Machine Interfaces (eHMIs) to communicate with other road users. Existing research primarily focuses on pedestrian-AV interactions, with limited attention given to other road users, such as cyclists and drivers of manually driven vehicles. So far, no studies have compared the effects of eHMIs across these three road user roles. Therefore, we conducted a within-subjects virtual reality experiment (N=40), evaluating the subjective and objective impact of an eHMI communicating the AV's intention to pedestrians, cyclists, and drivers under various levels of distraction (no distraction, visual noise, interference). eHMIs positively influenced safety perceptions, trust, perceived usefulness, and mental demand across all roles. While distraction and road user roles showed significant main effects, interaction effects were only observed in perceived usability. Thus, a unified eHMI design is effective, facilitating the standardization and broader adoption of eHMIs in diverse traffic.2026MCMark Colley et al.Ulm UniversityExternal HMI (eHMI) — Communication with Pedestrians & CyclistsCHI
Introducing ROADS: A Systematic Comparison of Remote Control Interaction Concepts for Automated Vehicles at Road WorksAs vehicle automation technology continues to mature, there is a necessity for robust remote monitoring and intervention features. These are essential for intervening during vehicle malfunctions, challenging road conditions, or in areas that are difficult to navigate. This evolution in the role of the human operator—from a constant driver to an intermittent teleoperator—necessitates the development of suitable interaction interfaces. While some interfaces were suggested, a comparative study is missing. We designed, implemented, and evaluated three interaction concepts (path planning, trajectory guidance, and waypoint guidance) with up to four concurrent requests of automated vehicles in a within-subjects study with N=23 participants. The results showed a clear preference for the path planning concept. It also led to the highest usability but lower satisfaction. With trajectory guidance, the fewest requests were resolved. The study’s findings contribute to the ongoing development of HMIs focused on the remote assistance of automated vehicles.2025MCMark Colley et al.Ulm University; UCL Interaction CentreAutomated Driving Interface & Takeover DesignTeleoperated DrivingCHI
Light My Way. Developing and Exploring a Multimodal Interface to Assist People With Visual Impairments to Exit Highly Automated VehiclesThe introduction of Highly Automated Vehicles (HAVs) has the potential to increase the independence of blind and visually impaired people (BVIPs). However, ensuring safety and situation awareness when exiting these vehicles in unfamiliar environments remains challenging. To address this, we conducted an interactive workshop with N=5 BVIPs to identify their information needs when exiting an HAV and evaluated three prior-developed low-fidelity prototypes. The insights from this workshop guided the development of PathFinder, a multimodal interface combining visual, auditory, and tactile modalities tailored to BVIP's unique needs. In a three-factorial within-between-subject study with N=16 BVIPs, we evaluated PathFinder against an auditory-only baseline in urban and rural scenarios. PathFinder significantly reduced mental demand and maintained high perceived safety in both scenarios, while the auditory baseline led to lower perceived safety in the urban scenario compared to the rural one. Qualitative feedback further supported PathFinder's effectiveness in providing spatial orientation during exiting.2025LMLuca-Maxim Meinhardt et al.Institute of Media Informatics, Ulm UniversityIn-Vehicle Haptic, Audio & Multimodal FeedbackVisual Impairment Technologies (Screen Readers, Tactile Graphics, Braille)CHI
Improving External Communication of Automated Vehicles Using Bayesian OptimizationThe absence of a human operator in automated vehicles (AVs) may require external Human-Machine Interfaces (eHMIs) to facilitate communication with other road users in uncertain scenarios, for example, regarding the right of way. Given the plethora of adjustable parameters, balancing visual and auditory elements is crucial for effective communication with other road users. With N=37 participants, this study employed multi-objective Bayesian optimization to enhance eHMI designs and improve trust, safety perception, and mental demand. By reporting the Pareto front, we identify optimal design trade-offs. This research contributes to the ongoing standardization efforts of eHMIs, supporting broader adoption.2025MCMark Colley et al.Ulm University; UCL Interaction CentreExternal HMI (eHMI) — Communication with Pedestrians & CyclistsExplainable AI (XAI)CHI
OptiCarVis: Improving Automated Vehicle Functionality Visualizations Using Bayesian Optimization to Enhance User ExperienceAutomated vehicle (AV) acceptance relies on their understanding via feedback. While visualizations aim to enhance user understanding of AV's detection, prediction, and planning functionalities, establishing an optimal design is challenging. Traditional "one-size-fits-all" designs might be unsuitable, stemming from resource-intensive empirical evaluations. This paper introduces OptiCarVis, a set of Human-in-the-Loop (HITL) approaches using Multi-Objective Bayesian Optimization (MOBO) to optimize AV feedback visualizations. We compare conditions using eight expert and user-customized designs for a Warm-Start HITL MOBO. An online study (N=117) demonstrates OptiCarVis efficacy in significantly improving trust, acceptance, perceived safety, and predictability without increasing cognitive load. OptiCarVis facilitates a comprehensive design space exploration, enhancing in-vehicle interfaces for optimal passenger experiences and broader applicability.2025PJPascal Jansen et al.Ulm University, Institute of Media InformaticsHead-Up Display (HUD) & Advanced Driver Assistance Systems (ADAS)AI-Assisted Decision-Making & AutomationCHI
Scrolling in the Deep: Analysing Contextual Influences on Intervention Effectiveness during Infinite Scrolling on Social MediaInfinite scrolling on social media platforms is designed to encourage prolonged engagement, leading users to spend more time than desired, which can provoke negative emotions. Interventions to mitigate infinite scrolling have shown initial success, yet users become desensitized due to the lack of contextual relevance. Understanding how contextual factors influence intervention effectiveness remains underexplored. We conducted a 7-day user study (N=72) investigating how these contextual factors affect users' reactance and responsiveness to interventions during infinite scrolling. Our study revealed an interplay, with contextual factors such as being at home, sleepiness, and valence playing significant roles in the intervention's effectiveness. Low valence coupled with being at home slows down the responsiveness to interventions, and sleepiness lowers reactance towards interventions, increasing user acceptance of the intervention. Overall, our work contributes to a deeper understanding of user responses toward interventions and paves the way for developing more effective interventions during infinite scrolling.2025LMLuca-Maxim Meinhardt et al.Institute of Media Informatics, Ulm UniversityNotification & Interruption ManagementCHI
Bumpy Ride? Understanding the Effects of External Forces on Spatial Interactions in Moving VehiclesAs the use of Head-Mounted Displays in moving vehicles increases, passengers can immerse themselves in visual experiences independent of their physical environment. However, interaction methods are susceptible to physical motion, leading to input errors and reduced task performance. This work investigates the impact of G-forces, vibrations, and unpredictable maneuvers on 3D interaction methods. We conducted a field study with 24 participants in both stationary and moving vehicles to examine the effects of vehicle motion on four interaction methods: (1) Gaze\&Pinch, (2) DirectTouch, (3) Handray, and (4) HeadGaze. Participants performed selections in a Fitts' Law task. Our findings reveal a significant effect of vehicle motion on interaction accuracy and duration across the tested combinations of Interaction Method $\times$ Road Type $\times$ Curve Type. We found a significant impact of movement on throughput, error rate, and perceived workload. Finally, we propose future research considerations and recommendations on interaction methods during vehicle movement.2025MSMarkus Sasalovici et al.Mercedes-Benz Tech Motion GmbH; Ulm University, Institute of Media InformaticsHead-Up Display (HUD) & Advanced Driver Assistance Systems (ADAS)Motion Sickness & Passenger ExperienceCHI
Fly Away: Evaluating the Impact of Motion Fidelity on Optimized User Interface Design via Bayesian Optimization in Automated Urban Air Mobility SimulationsAutomated Urban Air Mobility (UAM) can improve passenger transportation and reduce congestion, but its success depends on passenger trust. While initial research addresses passengers' information needs, questions remain about how to simulate air taxi flights and how these simulations impact users and interface requirements. We conducted a between-subjects study (N=40), examining the influence of motion fidelity in Virtual-Reality-simulated air taxi flights on user effects and interface design. Our study compared simulations with and without motion cues using a 3-Degrees-of-Freedom motion chair. Optimizing the interface design across six objectives, such as trust and mental demand, we used multi-objective Bayesian optimization to determine the most effective design trade-offs. Our results indicate that motion fidelity decreases users' trust, understanding, and acceptance, highlighting the need to consider motion fidelity in future UAM studies to approach realism. However, minimal evidence was found for differences or equality in the optimized interface designs, suggesting personalized interface designs.2025LMLuca-Maxim Meinhardt et al.Institute of Media Informatics, Ulm UniversityAutomated Driving Interface & Takeover DesignMotion Sickness & Passenger ExperienceCHI
Effects of Uncertain Trajectory Prediction Visualization in Highly Automated Vehicles on Trust, Situation Awareness, and Cognitive LoadColley 等人研究高度自动驾驶汽车中不确定轨迹预测可视化形式对驾驶员信任度、态势感知和认知负荷的影响机制。2024MCMark Colley et al.Automated Driving Interface & Takeover DesignExplainable AI (XAI)UbiComp
Introducing AV-Sketch: An Immersive Participatory Design Tool for Automated Vehicle — Passenger InteractionIn the emerging automated vehicle (AV)—passenger interaction domain, there is no agreed-upon set of methods to design early concepts. Non-designers may find it challenging to brainstorm interfaces for unfamiliar technology like AVs. Therefore, we explore using an immersive virtual environment to enable expert and non-expert designers to actively participate in the design phases. We built AV-Sketch, an in-situ (on-site) simulator that allows the creation of automotive interfaces while being immersed in VR depicting diverse AV-passenger interactions. At first, we conducted a participatory design study (𝑁=15) by utilizing PICTIVE (Plastic Interface for Collaborative Technology) to conceptualize human-machine interfaces for AV passengers. The findings led to the design of AV-Sketch, which we tested in a design session (𝑁=10), assessing users’ design experiences. Overall, participants felt more engaged and confident with the in-situ experience, enabling better contextualization of design ideas in real-world scenarios, with improved spatial considerations and dynamic aspects of in-vehicle interfaces.2024AAAshratuz Zavin Asha et al.Automated Driving Interface & Takeover DesignSocial & Collaborative VRAutoUI
Exploring Passenger-Automated Vehicle Negotiation Utilizing Large Language Models for Natural InteractionAs vehicle automation advances to SAE Levels 3 to 5, transitioning driving control from human to system, ensuring automated vehicles (AVs) align with user preferences becomes a challenge. Natural interaction emerges as a common goal, offering ways to convey user interests in a user-friendly manner. However, technical, legal, or design constraints may prevent fulfilling these preferences, leading to potential conflicts. Through an online survey (N=50), potential driver-passenger conflicts and their handling strategies were explored. Subsequently, in a Virtual Reality study (N=14), we applied identified strategies (ranging from distracting to motivating and adhering to social norms) to user-AV interactions using a state-of-the-art language model (GPT-4 Turbo) primed with the strategies to simulate realistic dialogues. Additionally, adaptive communication was compared to non-adaptive communication. Our findings reveal a preference for adaptive communication. Yet, despite using advanced modeling, accurately predicting user interactions remained challenging, with users often trying to outsmart the AI.2024ASAnnika Stampf et al.Automated Driving Interface & Takeover DesignHuman-LLM CollaborationAI-Assisted Decision-Making & AutomationAutoUI
Accessible Maps for the Future of Inclusive RidesharingFor people who are blind and low vision (BLV), ridesharing provides an important means of independence and mobility. However, a common challenge relates to finding the vehicle when it arrives to an unanticipated location. Although coordinating with the driver for assistance is serviceable in the near term, new solutions are necessary when a human is no longer available in future automated vehicles. Therefore, this paper presents and evaluates a multisensory smartphone-based map system designed to enable nonvisual tracking of summoned vehicles. Results from a user study with (N=12) BLV users suggest that vibro-audio maps (VAMs) promote superior spatial confidence and reasoning compared to current nonvisual audio interfaces in ridesharing apps, while also being desirable and easy to use. A subsequent expert evaluation based on improvements suggested during the user study indicate the practical utility of VAMs to address both current and future wayfinding challenges for BLV travelers.2024PFPaul D. S. Fink et al.External HMI (eHMI) — Communication with Pedestrians & CyclistsVisual Impairment Technologies (Screen Readers, Tactile Graphics, Braille)Ridesharing PlatformsAutoUI
Investigating the Effects of External Communication and Platoon Behavior on Manual Drivers at Highway AccessAutomated vehicles are expected to improve traffic safety and efficiency. One approach to achieve this is via platooning, that is, (automated) vehicles can drive behind each other at very close proximity to reduce air resistance. However, this behavior could lead to difficulties in mixed traffic, for example, when manual drivers try to enter a highway. Therefore, we report the results of a within-subject Virtual Reality study (N=29) evaluating different platoon behaviors (single vs. multiple, i.e., four, gaps) and communication strategies (HUD, AR, attached displays). Results show that AR communication reduced mental workload, improved perceived safety, and a single big gap led to the safest merging behavior. Our work helps to incorporate novel behavior enabled by automation into general traffic better.2024MCMark Colley et al.Ulm UniversityExternal HMI (eHMI) — Communication with Pedestrians & CyclistsHead-Up Display (HUD) & Advanced Driver Assistance Systems (ADAS)CHI
Scalability in External Communication of Automated Vehicles: Evaluation and Recommendations"Automated vehicles will alter traffic fundamentally. While users can engage in non-driving-related tasks such as reading or even sleeping, the possibility to interact with other road users such as pedestrians via, for example, eye contact vanishes. Therefore, external communication of automated vehicles is currently researched with various concepts spanning dimensions such as anthropomorphism, technology, viewpoint, locus, message type, and others. However, the proposed concepts are mostly evaluated in simple scenarios, such as one person trying to cross in front of one automated vehicle. Therefore, we implemented a WebGL application of a four-lane road and conducted a within-subject study (N=46) to study the effects of nine concepts with and without the presence of other pedestrians and altering the yielding target of the automated vehicle. We found that all concepts were rated better than having no external communication. However, the effects were not uniform across the concepts. https://doi.org/10.1145/3596248"2023MCMark Colley et al.External HMI (eHMI) — Communication with Pedestrians & CyclistsUbiComp
Come Fly With Me - Investigating the Effects of Path Visualizations in Automated Urban Air Mobility"Automated Urban Air Mobility will enhance passenger transportation in metropolitan areas in the near future. Potential passengers, however, have little knowledge about this mobility form. Therefore, there could be concerns about safety and low trust. As trajectories are essential information to address these concerns, we evaluated seven path visualizations in an online video-based study (N=99). We found that a path line visualization was rated highest for trust and perceived safety. In a follow-up virtual reality study (N=24), we evaluated the effects of this visualization and of other air traffic flying by. We found that the participants looked at the path line more often when other air traffic was present and that the path line increased trust and predictability of the air taxi's future path. https://doi.org/10.1145/3596249"2023MCMark Colley et al.AR Navigation & Context AwarenessPublic Transit & Trip PlanningUbiComp
The Loop and How to Break It: Investigating Infinite Scrolling Behaviour in Social Media Applications and Reasons to StopToday's social media (SM) platforms are toolkits consisting of features with different use cases, some strongly related to habitual and regretful use. Especially Infinite Scrolling (IS) has been reported to make users feel like they are being caught in a loop, regretfully elongating SM sessions. We investigated and defined this loop while unveiling the processes that make users break it. Based on a one-week-long field study (N=46), we unfolded and categorized general reasons for leaving social media and related those to IS. In light of our findings, we argue that interventions - tackling regretful use - should not only focus on the app but incorporate the user's context, as most reasons to break SM sessions were not related to the app but the user's general context. Our findings and prior work also indicate the coexistence of multiple loops, which we define as inner (intra-session) loops surrounded by an outer (habitual) loop.2023JRJan Ole Rixen et al.Social Platform Design & User BehaviorCyberbullying & Online HarassmentMobileHCI
Effects of Urgency and Cognitive Load on Modality Usage in Highly Automated VehiclesIn highly automated vehicles, passengers can engage in non-driving-related activities. Additionally, the technical advancement allows for novel interaction possibilities such as voice, gesture, gaze, touch, or multimodal interaction, both to refer to in-vehicle and outside objects (e.g., thermostat or restaurant). This interaction can be characterized by levels of urgency (e.g., based on late detection of objects) and cognitive load (e.g., because of watching a movie or working). Therefore, we implemented a Virtual Reality simulation and conducted a within-subjects study with N=11 participants evaluating the effects of urgency and cognitive load on modality usage in automated vehicles. We found that while all modalities were possible to use, participants relied on touch the most. This was followed by gaze, especially for external referencing. This work helps to further understand multimodal interaction and the requirements this poses on natural interaction in (automated) vehicles.2023MCMark Colley et al.Automated Driving Interface & Takeover DesignExternal HMI (eHMI) — Communication with Pedestrians & CyclistsIn-Vehicle Haptic, Audio & Multimodal FeedbackMobileHCI
I've Got the Power: Exploring the Impact of Cooperative Systems on Driver-initiated Takeovers and Trust in Automated Vehicles in Conflicting SituationsDrivers want to retain a sense of control when driving (partially) automated vehicles (AVs). Future AVs will continue to offer the possibility to drive manually, potentially leading to challenging driver-initiated takeovers (DITs) due to the "out-of-the-loop problem" and reduced driving performance. A driving simulator study (N=24) was conducted to explore whether cooperative systems, without full control of driving tasks, provide a sense of control to mitigate DITs in varying conflict situations. Conflict levels were operationalized by an AV performing overtaking maneuvers under free, 100m, and 50m visibility on a two-lane rural road. Participants experienced three systems: no intervention-, a cooperative choice-, and a manual control system. Results showed that participants had a similar sense of control with the cooperative system compared to the manual one and preferred it over the manual system. The likelihood of DITs increased with conflict intensity, and trust in the AV moderated the conflict-DIT association.2023MWMarcel Woide et al.Automated Driving Interface & Takeover DesignAI-Assisted Decision-Making & AutomationAutoUI
AdVANcing Design: Customizing Spaces for VanlifeThis study examines three modalities for designing live-in van interiors. Participants (N=18) situated within an empty van were asked to explore potential designs using physical cardboard prototyping, a commercial software application (Vanspace 3D) for planning van interiors, and an augmented reality application that we developed. Participants were asked to think aloud as they designed van interiors for fictive journeys using each modality. A qualitative evaluation was conducted to assess how participants' conceptualizations of space shifted across the use of each prototyping method. The results demonstrate that each design method influenced design outcomes due to the physicality of the task. This study highlights the importance of considering the role of physicality in the selection of prototyping modality for the design exploration process.2023SSSaki Suzuki et al.Shape-Changing Interfaces & Soft Robotic MaterialsShape-Changing Materials & 4D PrintingCustomizable & Personalized ObjectsAutoUI
Interaction Effects of Pedestrian Behavior, Smartphone Distraction and External Communication of Automated Vehicles on Crossing and Gaze BehaviorExternal communication of automated vehicles is proposed to replace driver-pedestrian communication in ambiguous crossing situations. So far, research has focused on simpler scenarios with one attentive pedestrian and one automated vehicle. This virtual reality study (N=115) investigates a more complex scenario with other crossing pedestrians, a distracting task on the smartphone, and external communication by the automated vehicle. Interaction effects were found for crossing duration, gaze behavior, and subjective measures. For attentive pedestrians, the external communication resulted in shorter crossing durations, higher perceived safety, as well as lower perceived criticality, cognitive workload, and effort. These positive effects were not found when pedestrians were distracted. Instead, distracted pedestrians benefited from other crossing pedestrians because they looked less at the stopping vehicle, felt safer, perceived the situation as less critical, and reported lower cognitive workload and effort. Pedestrians initiated crossings earlier with a group or external communication and later with a smartphone.2023MLMirjam Lanzer et al.Ulm UniversityExternal HMI (eHMI) — Communication with Pedestrians & CyclistsTeleoperated DrivingCHI