Evaluating Generative AI in the Lab: Methodological Challenges and GuidelinesGenerative AI (GenAI) systems are inherently non-deterministic, producing varied outputs even for identical inputs. While this variability is central to their appeal, it challenges established HCI evaluation practices that typically assume consistent and predictable system behavior. Designing controlled lab studies under such conditions therefore remains a key methodological challenge. We present a reflective multi-case analysis of four lab-based user studies with GenAI-integrated prototypes, spanning conversational in-car assistant systems and image generation tools for design workflows. Through cross-case reflection and thematic analysis across all study phases, we identify five methodological challenges and propose eighteen practice-oriented recommendations, organized into five guidelines. These challenges represent methodological constructs that are either amplified, redefined, or newly introduced by GenAI's stochastic nature: (C1) reliance on familiar interaction patterns, (C2) fidelity-control trade-offs, (C3) feedback and trust, (C4) gaps in usability evaluation, and (C5) interpretive ambiguity between interface and system issues. Our guidelines address these challenges through strategies such as reframing onboarding to help participants manage unpredictability, extending evaluation with constructs such as trust and intent alignment, and logging system events, including hallucinations and latency, to support transparent analysis. This work contributes (1) a methodological reflection on how GenAI's stochastic nature unsettles lab-based HCI evaluation and (2) eighteen recommendations that help researchers design more transparent, robust, and comparable studies of GenAI systems in controlled settings.2026HPHyerim Park et al.BMW GroupGenerative AI (Text, Image, Music, Video)Human-LLM CollaborationUser Research Methods (Interviews, Surveys, Observation)IUI
Enhancing Generative AI Image Refinement with Scribbles and Annotations: A Comparative Study of Multimodal PromptsGenerative AI (GenAI) image tools are increasingly used in design practice, enabling rapid ideation but offering limited support for refinement tasks such as adjusting layout, scale, or visual attributes. While text prompts and inpainting allow localized edits, they often remain inefficient or ambiguous for precise, in-context, and iterative refinement---motivating the exploration of alternative methods. This work examines how pen-based scribbles and annotations can enhance GenAI image refinement. A formative study with seven professional designers informed a prototype supporting three input modalities: text-only, visual-only, and combined prompting. A within-subjects study with 30 designers and design students compared these modalities across closed- and open-ended tasks, evaluating expressiveness, efficiency, workload, user experience, iteration, and multimodal strategies. Visual prompts improved clarity and speed for spatial edits while reducing workload, whereas text remained effective for semantic and global changes. The combined modality received the highest overall ratings, enabling complementary use, balancing spatial precision with semantic detail, and supporting smoother iteration. Task-specific preferences also emerged: adding new objects often required both modalities, while moving or modifying elements was typically handled through visual input. This work contributes (1) an empirical comparison of multimodal prompting for GenAI refinement, (2) a prototype integrating scribbles and annotations, and (3) insights into designers' multimodal strategies to inform future GenAI interfaces that better support refinement in GenAI-supported design workflows.2026HPHyerim Park et al.BMW GroupGenerative AI (Text, Image, Music, Video)Creative Collaboration & Feedback SystemsGraphic Design & Typography ToolsIUI
Interaction Methods in Generative AI Image Tools: A Review of Trends and Design Opportunities Across HCI and IndustryGenerative AI (GenAI) image tools are increasingly integrated into design workflows, prompting HCI research on their interaction methods and interfaces. We reviewed 37 such tools, including 28 HCI research systems and nine commercial systems (2022--July 2025), using three analytical frameworks: interaction methods, creative processes, and tool functionalities. We found that text prompts remain the dominant input method, while visual and attribute-based inputs---particularly in academic tools---are gaining traction and are often combined with text for refinement. Commercial systems emphasize parameter control, whereas academic tools focus on semantic attributes and visual organization. Most tools support ideation and exploration, but provide limited support for refinement and evaluation. Based on these findings, we identify nine design opportunities, including advanced visual interaction, simplified parameter control, precision editing, direct manipulation, workflow integration, default settings that support rapid exploration, and user guidance for later stages. We contribute a framework for analyzing GenAI interfaces and actionable directions for designing more usable, creativity-supportive GenAI image systems.2026HPHyerim Park et al.University of StuttgartGenerative AI (Text, Image, Music, Video)Human-LLM CollaborationCreative Collaboration & Feedback SystemsCHI
"What Are You Doing?": Effects of Intermediate Feedback from Agentic LLM In-Car Assistants During Multi-Step ProcessingAgentic AI assistants that autonomously perform multi-step tasks raise open questions for user experience: how should such systems communicate progress and reasoning during extended operations, especially in attention-critical contexts such as driving? We investigate feedback timing and verbosity from agentic LLM-based in-car assistants through a controlled, mixed-methods study (N=45) comparing planned steps and intermediate results feedback against silent operation with final-only response. Using a dual-task paradigm with an in-car voice assistant, we found that intermediate feedback significantly improved perceived speed, trust, and user experience while reducing task load - effects that held across varying task complexities and interaction contexts. Interviews further revealed user preferences for an adaptive approach: high initial transparency to establish trust, followed by progressively reducing verbosity as systems prove reliable, with adjustments based on task stakes and situational context. These findings inform design principles for feedback systems in agentic AI assistants, balancing transparency and efficiency across domains.2026JKJohannes Kirmayr et al.BMW GroupVoice User Interface (VUI) DesignIntelligent Voice Assistants (Alexa, Siri, etc.)Head-Up Display (HUD) & Advanced Driver Assistance Systems (ADAS)CHI
Unraveling Subjective ADAS Comprehension Considering Factors of Situational Complexity on the Example of Traffic Light ScenariosAdvanced driver assistance systems (ADAS) with increasing automation maturity and availability in urban contexts are entering the market. Meanwhile, the situational context has been identified to play a crucial role in system comprehension and usage, yet its subcomponents and their relation to system comprehension remain an open research question. To gain insights in the role of the situation complexity regarding subjective system comprehension and different methodological aspects, this study applies a mixed quantitative and qualitative approach, focusing on signaled intersections as an exemplary scenario. An on-road study with forty-six participants was conducted, involving six traffic light scenarios (all experienced twice). Results indicate that while comprehension was generally high, the situational context, including environmental and traffic-related factors, affected subjective system understanding. The proposed approach sheds light on the role of mixed methods in ADAS research, which may provide insights for system developers and suggestions for user training content.2025CBClaudia Buchner et al.Head-Up Display (HUD) & Advanced Driver Assistance Systems (ADAS)AutoUI
Digital Forms for All: A Holistic Multimodal Large Language Model Agent for Health Data EntryCuadra 等人提出多模态大语言模型代理，整合视觉、语音和文本输入实现自动化健康数据录入，降低表单填写负担。2024ACAndrea Cuadra et al.Human-LLM CollaborationMotor Impairment Assistive Input TechnologiesMental Health Apps & Online Support CommunitiesUbiComp
PASTEL: Privacy-Preserving Federated Learning in Edge ComputingElhattab 等人提出 PASTEL 隐私保护联邦学习方案，应用于边缘计算环境。2024FEFatima Elhattab et al.Generative AI (Text, Image, Music, Video)Privacy by Design & User ControlUbiComp
Exploring Urban Challenges: Understanding Advanced Driver Assistance Systems in Different Situational ContextsNew Advanced Driver Assistance Systems (ADAS) are now available to support urban driving. To adequately use ADAS, especially in complex situations, drivers must comprehend them. An on-road study was conducted to investigate the mental model development while interacting with a state-of-the-art ADAS in both a rural (less complex) and an urban context (more complex). Forty-six participants experienced two rounds of each context. After each round, drivers rated their mental model, acceptance, and trust. Results indicate that for the rural context participants learned the system functionality in the first round without further improvement . In the urban context the mental model was generally less accurate, but improved in the second round. Trust increased from the first to the second rural round while acceptance did not show a significant change within the context. The results provide a first glimpse into the importance of evaluating different contexts and interaction scenarios for ADAS.2024CBClaudia Buchner et al.Head-Up Display (HUD) & Advanced Driver Assistance Systems (ADAS)AutoUI
Improving Driver Engagement with Level 2 Automated Systems: The Impact of Fully Shared Longitudinal ControlAccording to the Society of Automotive Engineers (SAE), in Level 2 systems (L2 systems), the system executes the longitudinal and lateral control of the vehicle, with the driver required to monitor the environment and intervene when necessary. To further improve safety and driver engagement, we compared a fully shared longitudinal control system, which permits speed adjustments via acceleration and braking without deactivation, with a conventional system that disengages upon braking. In a simulator study involving 61 participants, both systems were well-received in terms of acceptance and user experience. The fully shared longitudinal control led to more frequent and earlier braking, suggesting anticipatory driving, without compromising perceived safety. Furthermore, it outperformed in hedonic qualities of user experience, and elicited a stronger intention to use. Our findings indicate that fully shared longitudinal control can enhance driver engagement, offering a valuable improvement for L2 automated systems.2024JIJohannes Illgner et al.Automated Driving Interface & Takeover DesignHead-Up Display (HUD) & Advanced Driver Assistance Systems (ADAS)AutoUI
Driven to Distraction: Exploring Mind Wandering During a Virtual Reality City DriveResearch has characterized mind-wandering as humans' natural mental state, with moments of task-focused attention being the exception. With this framing, mind-wandering while driving likely occurs more than generally acknowledged, and seems poised to increase with higher levels of automation. This in turn may have adverse effects on drivers' abilities to regain situation awareness or resume control when needed. Of the prior work on detecting mind-wandering while driving, none focuses on automation or complex urban environments. We ran an exploratory study (N=14) of an automated drive through New York City in a two-dimensional virtual reality context, focusing on physiological measures such as gaze distribution, pupillometry, and heart rate. We also explored how drivers missing critical events may be a potential new measure. Results varied between focused and mind-wandering mental states and between moving and stopped driving contexts. These observations are an initial step toward understanding mind-wandering across diverse driving scenarios.2024RCRebecca Currano et al.Automated Driving Interface & Takeover DesignEye Tracking & Gaze InteractionAutoUI
Listening to the Voices: Describing Ethical Caveats of Conversational User Interfaces According to Experts and Frequent UsersAdvances in natural language processing and understanding have led to a rapid growth in the popularity of conversational user interfaces (CUIs). While CUIs introduce novel benefits, they also yield risks that may exploit people's trust. Although research looking at unethical design deployed through graphical user interfaces (GUIs) established a thorough taxonomy of so-called dark patterns, there is a need for an equally in-depth understanding in the context of CUIs. Addressing this gap, we interviewed 27 participants from three cohorts: researchers, practitioners, and frequent users of CUIs. Applying thematic analysis, we develop five themes reflecting each cohort's insights about ethical design challenges and introduce the CUI Expectation Cycle, bridging system capabilities and user expectations while respecting each theme's ethical caveats. This research aims to inform future work to consider ethical constraints while adopting a human-centred approach.2024TMThomas Mildner et al.University of BremenVoice User Interface (VUI) DesignAI Ethics, Fairness & AccountabilityDark Patterns RecognitionCHI
Investigating Hazard Notifications for Cyclists in Mixed Reality: A Comparative Analysis with a Test Track StudyOne way to improve road safety for cyclists is the development of hazard notification systems. Instead of in field experiments, such systems could be tested in safe and more controlled simulated environments; however, their validity needs verification. We evaluated the validity of mixed reality (MR) simulation for bicycle support systems notifying of dooring hazards. In a mixed-design study (N=43) with environment type(MR/test track) as within and hazard notifications (with/without) as between factor, comparing subjective and objective measures across environments. In conclusion, MR simulation is absolutely valid for user experience and perceived safety and relatively valid for workload, standard deviation of lateral position, and speed. However, MR simulation was not valid for lateral distance, as participants cycled more in the center of the street than on the test track, perhaps to avoid simulator sickness. Thus, we conclude that MR simulation is valuable for studying bicycle safety.2023TSTamara von Sawitzky et al.External HMI (eHMI) — Communication with Pedestrians & CyclistsV2X (Vehicle-to-Everything) Communication DesignAutoUI
Development of a Perceived Security Scale for Shared Automated Vehicles (PSSAVS) and its Validation in Colombia and GermanyPerceived security is crucial for the widespread adoption of shared automated vehicles (SAVs) and shuttle buses. However, there is currently no validated instrument to measure perceived security in this context, and little research has been done to determine the factors that contribute to perceived security. We propose the Perceived Security Scale for Shared Automated Vehicles (PSSAV), a questionnaire that assesses various aspects of perceived security in SAVs. The scale was evaluated using an exploratory, data-driven approach in a pilot study with 60 German participants, and a main study with 114 German and 101 Colombian participants experiencing a positive or negative ride in an automated shuttle bus (between-subjects design) presented as videos in an online study. The results suggest that trust, privacy, and control are key factors that influence security in the context of SAVs. The PSSAV questionnaire is reliable and sensitive to manipulation, indicating its construct validity.2023MSMartina Schuß et al.Automated Driving Interface & Takeover DesignMotion Sickness & Passenger ExperienceAutoUI
Investigating Various Dynamics in the Box Task Combined With a Detection Response Task: Are There Performance Differences Between Uniform and Non-uniform Box Dynamics?The Box Task combined with a Detection Response Task (BT + DRT) is a relatively less investigated but promising method for evaluating visual-manual and cognitive task demand due to the interaction with in-vehicle information systems while driving. The BT includes the tracking of a dynamic box whose size and position follow a sinusoidal pattern with uniform amplitudes and frequencies. However, it is unclear whether participants are able to predict and adapt to these uniform dynamics, which might lead to a reduced sensitivity of the BT + DRT. Within the present study, it was aimed to examine differences in BT + DRT performance depending on uniform and non-uniform BT dynamics. A laboratory study was conducted with N = 41 participants. The experimental conditions differed in the type and difficulty level of the secondary tasks as well as in the BT dynamics (uniform, varying amplitude, varying frequency). While the uniform BT dynamics could be more predictable, the non-uniform BT dynamics were designed slightly easier in their difficulty using a lower frequency or amplitude. The results revealed no performance benefits when performing uniform BT dynamics compared to non-uniform BT dynamics. The frequency BT condition was related to a significantly lower variability of box position and higher gaze duration on the secondary task compared to the uniform BT dynamics. These findings suggest that participants are not or only negligible able to adapt to the uniform BT dynamics. Therefore, it is recommended to use the uniform BT dynamics as suggested and implemented in previous studies.2023DTDaniel Trommler et al.Head-Up Display (HUD) & Advanced Driver Assistance Systems (ADAS)Eye Tracking & Gaze InteractionAutoUI
Multimodal Fusion for Driver Referencing: A Comparison of Pointing to Objects Inside and Outside the VehicleAdvanced in-cabin sensing technologies, especially vision based approaches, have tremendously progressed user interaction inside the vehicle, paving the way for new applications of natural user interaction. Just as humans use multiple modes to communicate with each other, we follow an approach which is characterized by simultaneously using multiple modalities to achieve natural human machine interaction for a specific task: pointing to or glancing towards objects inside as well as outside the vehicle for deictic references. By tracking the movements of eye-gaze, head and finger, we design a multimodal fusion architecture using a deep neural network to precisely identify the driver's referencing intent. Additionally, we use a speech command as a trigger to separate each referencing event. We observe differences in driver behavior in the two pointing use cases (i.e. for inside and outside objects), especially when analyzing the preciseness of the three modalities eye, head, and finger. We conclude that there is no single modality that is solely optimal for all cases as each modality reveals certain limitations. Fusion of multiple modalities exploits the relevant characteristics from each modality, hence overcoming the case dependent limitations of each individual modality. Ultimately, we propose a method to identity whether the driver's intended object lies inside or outside the vehicle, based on the predicted pointing direction.2022AAAbdul Rafey Aftab et al.In-Vehicle Haptic, Audio & Multimodal FeedbackHand Gesture RecognitionVoice User Interface (VUI) DesignIUI
Measuring user experience in automated driving: Developing a single-item measure Measuring user experience is highly important for human-centered development and thus for designing automated driving systems. Multi-item measures such as the System Usability Scale (SUS) or the Usability Metric for User Experience (UMUX) are commonly used for collecting user feedback on technical systems or products. The goal of the present study was to investigate the potentials of a single-item approach as an economic alternative for measuring user experience compared to multi-item scales. Therefore, a single-item measure was developed to assess both event-related and cumulative user experience in automated driving. User experience was manipulated in a between-subject design implemented in a real-world driving task and feedback was collected using the newly developed Single Item User Experience (SIUX) scale, the UMUX, and the SUS. Results indicate that the SIUX scale is more sensitive than the UMUX to differences in event-related user experience, but not in cumulative user experience. Both the SIUX and the UMUX were more sensitive than the SUS when measuring differences in cumulative user experience. Future studies should be aimed at investigating the applicability of the SIUX scale to domains other than automated driving and at collecting more extensive data on validity and reliability of all three instruments.2021CHChantal Himmels et al.Automated Driving Interface & Takeover DesignPrivacy by Design & User ControlUser Research Methods (Interviews, Surveys, Observation)AutoUI
How to Design the Perfect Prompt: A Linguistic Approach to Prompt Design in Automotive Voice Assistants – An Exploratory StudyIn-vehicle voice user interfaces (VUIs) are becoming increasingly popular while needing to handle more and more complex functions. While many guidelines exist in terms of dialog design, a methodical and encompassing approach to prompt design is absent in the scientific landscape. The present work closes this gap by providing such an approach in form of linguistic-centered research. By extracting syntactical, lexical, and grammatical parameters from a German contemporary grammar, we examine how their respective manifestations affect users’ perception of a given system output across different prompt types. Through exploratory studies with a total of 1,206 participants, we provide concrete best practices to optimize and refine the design of VUI prompts. Based on these best practices, three superordinate user needs regarding prompt design can be identified: a) a suitable level of (in)formality, b) a suitable level of complexity/simplicity, and c) a suitable level of (im)mediacy.2021AMAnna-Maria Meck et al.Voice User Interface (VUI) DesignIntelligent Voice Assistants (Alexa, Siri, etc.)AutoUI
At Your Service: Designing Voice Assistant Personalities to Improve Automotive User Interfaces: A Real World Driving StudyThis paper investigates personalized voice characters for in-car speech interfaces. In particular, we report on how we designed different personalities for voice assistants and compared them in a real world driving study. Voice assistants have become important for a wide range of use cases, yet current interfaces are using the same style of auditory response in every situation, despite varying user needs and personalities.To close this gap, we designed four assistant personalities (Friend, Admirer, Aunt, and Butler) and compared them to a baseline (Default) in a between-subject study in real traffic conditions. Our results show higher likability and trust for assistants that correctly match the user's personality while we observed lower likability, trust, satisfaction, and usefulness for incorrectly matched personalities, each in comparison with the Default character. We discuss design aspects for voice assistants in different automotive use cases.2019MBMichael Braun et al.BMW Group Research, New Technologies, Innovations & Ludwig Maximilian University of MunichVoice User Interface (VUI) DesignAgent Personality & AnthropomorphismCHI
What makes an automated vehicle a good driver? Exploring lane change announcements in dense traffic situations.An automated vehicle needs to learn how human road users experience the intentions of other drivers and understand how they communicate with each other in order to avoid misunderstandings and prevent giving a negative external image during interactions. The aim of the present study is to identify a cooperative lane change indication which other drivers understand unambiguously and prefer when it comes to lane change announcements in a dense traffic situation on the highway. A fixed-base driving simulator study is conducted with N = 66 participants in Germany in a car-following scenario. Participants rated, from the lag driver’s perspective, different lane change announcements of another driver which varied in lateral movements (i.e., duration, lateral offset). Main findings indicate that a medium offset and moderate duration of lateral movement is experienced as most cooperative. The results are crucial for the development of lane change strategies for automated vehicles.2018NKNina Kauffmann et al.BMW GroupAutomated Driving Interface & Takeover DesignV2X (Vehicle-to-Everything) Communication DesignCHI
P3 - How Usability Can Save the Day – Methodological Considerations for Making Automated Driving a Success StoryIt will not be long until Level 3 Automated Driving Systems (L3 ADS) enter the consumer market. An important role corresponds to methodology development. The present paper gives impetus to the process of developing robust methods for evaluating Human-Machine Interfaces (HMI) for L3 ADS. First, a literature review on automotive interfaces concerning methodology application is outlined showing that studies often lack to provide both self-report and observational data. To derive a comprehensive image of HMI quality, we recommend multi-method approach in user research. Subsequently, we provide an overview of state-of-the-art self-report and observational measures. From the availability of measures and the necessity to include both in user studies, the issue of the performance-preference dissociation arises. It threatens study designs and interpretation of results. Following methodological recommendations from the present work supports researchers and practitioners in the area of automated driving for proper study design and interpretation of study results.2018YFYannick Forster et al.Automated Driving Interface & Takeover DesignAutoUI