Understanding Data Usage when Making High-Stakes Frontline Decisions in Homelessness ServicesFrontline staff of emergency shelters face challenges such as vicarious trauma, compassion fatigue, and burnout. The technology they use is often not designed to fit their unique needs and can feel burdensome, on top of already cognitively and emotionally taxing work. While existing literature focuses on data-driven technologies that automate or streamline frontline decision-making about vulnerable clients, we discuss scenarios when staff may resist such automation, and suggest how data-driven technologies can better align with their human-centred decision-making processes. This paper presents findings from a longitudinal qualitative fieldwork study conducted at a large emergency shelter in Canada, from 2022-2024. The goal of this fieldwork was to co-design, develop, and deploy an interactive data-navigation interface that supports frontline staff during collaborative, high-stakes decision-making about clients experiencing homelessness. Such decisions are high-stakes as they can have potential impacts on clients' survival. In reflecting on this fieldwork, we contribute insight into the role of administrative shelter data in these decisions, and unpack staff members' apparent reluctance to outsource decisions about vulnerable clients to data systems. We discuss the notion of a "data-outsourcing continuum" in terms of how future designers may create technologies that support compassionate data-driven decision-making in nonprofit, human-centred domains.2025TMTeale W Masrani et al.Humans vs. AI for Decision MakingCSCW
MapStory: Prototyping Editable Map Animations with LLM AgentsWe introduce MapStory, an LLM‑powered animation prototyping tool that generates editable map animation sequences directly from natural language text by leveraging a dual-agent LLM architecture. Given a user-written script, MapStory automatically produces a scene breakdown, which decomposes the text into key map animation primitives such as camera movements, visual highlights, and animated elements. Our system includes a researcher agent that accurately queries geospatial information by leveraging an LLM with web search, enabling automatic extraction of relevant regions, paths, and coordinates while allowing users to edit and query for changes or additional information to refine the results. Additionally, users can fine-tune parameters of these primitive blocks through an interactive timeline editor. We detail the system’s design and architecture, informed by formative interviews with professional animators and by an analysis of 200 existing map animation videos. Our evaluation, which includes expert interviews (N=5), and a usability study (N=12), demonstrates that MapStory enables users to create map animations with ease, facilitates faster iteration, encourages creative exploration, and lowers barriers to creating map-centric stories.2025AGAditya Gunturu et al.Geospatial & Map VisualizationComputational Methods in HCIUIST
FEDT: Supporting Experiment Design and Execution in HCI Fabrication ResearchFabrication research in HCI relies on diverse experiments to inform and assess research contributions. However, performance and reporting of these experiments is inconsistent, not only reducing transparency that reassures reviewers and readers of a project's rigour but also challenging methods' replicability by future researchers. We analyze recent fabrication publications to extract a unified experimental workflow, which we develop into a domain-specific language, and into the openly-available Fabrication Experiment Design Tool (FEDT). FEDT facilitates designing and executing HCI fabrication experiments. We demonstrate its comprehensiveness by using FEDT to model 42 fabrication experiments from 10 papers, which leverage varied fabrication technologies and techniques, including requiring human intervention in their steps. We discuss FEDT and our modeled experiments with the papers' original authors to evaluate its precision and utility in real workflows, and we demonstrate functionality with end-to-end replications of two published experiments.2025VSValkyrie Savage et al.Prototyping & User TestingComputational Methods in HCIResearch Ethics & Open ScienceUIST
SparseEMG: Computational Design of Sparse EMG Layouts for Sensing GesturesGesture recognition with electromyography (EMG) is a complex problem influenced by gesture sets, electrode count and placement, and machine learning parameters (e.g., features, classifiers). Most existing toolkits focus on streamlining model development but overlook the impact of electrode selection on classification accuracy. In this work, we present the first data-driven analysis of how electrode selection and classifier choice affect both accuracy and sparsity. Through a systematic evaluation of 28 combinations (4 selection schemes, 7 classifiers), across six datasets, we identify an approach that minimizes electrode count without compromising accuracy. The results show that Permutation Importance (selection scheme) with Random Forest (classifier) reduces the number of electrodes by 53.5%. Based on these findings, we introduce SparseEMG, a design tool that generates sparse electrode layouts based on user-selected gesture sets, electrode constraints, and ML parameters while also predicting classification performance. SparseEMG supports 50+ unique gestures and is validated in three real-world applications using different hardware setups. Results from our multi-dataset evaluation show that the layouts generated from the SparseEMG design tool are transferable across users with only minimal variation in gesture recognition performance.2025AKAnand Kumar et al.Electrical Muscle Stimulation (EMS)Hand Gesture RecognitionUIST
Guided Reality: Generating Visually-Enriched AR Task Guidance with LLMs and Vision ModelsLarge language models (LLMs) have enabled the automatic generation of step-by-step augmented reality (AR) instructions for a wide range of physical tasks. However, existing LLM-based AR guidance often lacks rich visual augmentations to effectively embed instructions into spatial context for a better user understanding. We present Guided Reality, a fully automated AR system that generates embedded and dynamic visual guidance based on step-by-step instructions. Our system integrates LLMs and vision models to: 1) generate multi-step instructions from user queries, 2) identify appropriate types of visual guidance, 3) extract spatial information about key interaction points in the real world, and 4) embed visual guidance in physical space to support task execution. Drawing from a corpus of user manuals, we define five categories of visual guidance and propose an identification strategy based on the current step. We evaluate the system through a user study (N=16), completing real-world tasks and exploring the system in the wild. Additionally, four instructors shared insights on how Guided Reality could be integrated into their training workflows.2025AZAda Yi Zhao et al.AR Navigation & Context AwarenessHuman-LLM CollaborationUIST
“It’s more of a vibe I’m going for”: Designing Text-to-Music Generation Interfaces for Video CreatorsBackground music plays a crucial role in social media videos, yet finding the right music remains a challenge for video creators. These creators, often not music experts, struggle to describe their musical goals and compare options. AI text-to-music generation presents an opportunity to address these challenges by allowing users to generate music through text prompts; however, these models often require musical expertise and are difficult to control. In this paper, we explore how to incorporate music generation into video editing workflows. A formative study with video creators revealed challenges in articulating and iterating on musical preferences, as creators described music as "vibes" rather than with explicit musical vocabulary. Guided by these insights, we developed a creative assistant for music generation using editable vibe-based recommendations and structured refinement of music output. A user study showed that the assistant supports exploration, while direct prompting is more effective for precise goals. Our findings offer design recommendations for AI music tools for video creators.2025NHNoor Hammad et al.Generative AI (Text, Image, Music, Video)Music Composition & Sound Design ToolsVideo Production & EditingDIS
AR-Based Embodied Avatar Assistance for Nonspeaking Autistic People? Design and Feasibility StudyMany nonspeaking autistic individuals rely on Communication and Regulation Partners (CRPs) to develop spelling-based communication using physical letterboards, but this support is often geographically inaccessible. We developed a remote presence system using Augmented Reality (AR) to enable immersive, collaborative spelling instruction. The system features holographic letterboards and fully embodied avatars with real-time head and hand tracking, allowing remote interaction between students and CRPs. In a study with 18 nonspeaking autistic participants, 15 (83%) successfully completed avatar-supported sessions. Interaction was higher, and participants reported a preference for the avatar condition over voice-only support. These findings demonstrate the feasibility of avatar-based AR telepresence for remote communication training. The system provides a demonstration of AR-supported interaction designed with nonspeaking autistic individuals—an underrepresented group in HCI—and offers design insights for inclusive telepresence technologies that address geographic and accessibility barriers.2025TDTravis Dow et al.Identity & Avatars in XRAugmentative & Alternative Communication (AAC)DIS
Grab-and-Release Spelling in XR: A Feasibility Study for Nonspeaking Autistic People Using Video-Passthrough DevicesThis paper explores the feasibility of using video-passthrough Extended Reality (XR) devices to support communication in nonspeaking autistic individuals. Prior XR work relied on expensive AR headsets and near-hand tapping interactions. We present LetterBox, a novel application for video-passthrough XR headsets (e.g., Meta Quest) that enables spelling via a “grab-snap-release” interaction. The app includes three immersion levels and a dynamic pass-through window to maintain caregiver presence. We conducted a study with 19 participants across four North American sites. All completed a multiphase spelling task and answered open-ended questions. Despite tolerability concerns, all participants wore the headset throughout; only one requested a break. The average spelling accuracy was 90.91%. In open-spelling, 14 participants responded—often independently. Reaction time and interaction speed data highlighted the impact of visual complexity, offering insights for reducing errors. These findings suggest video-passthrough XR is well tolerated and that grab-snap-release interactions may benefit users with motor challenges.2025LALorans Alabood et al.Identity & Avatars in XRSpecial Education TechnologyDIS
Video2MR: Automatically Generating Mixed Reality 3D Instructions by Augmenting Extracted Motion from 2D VideosThis paper introduces Video2MR, a mixed reality system that automatically generates 3D sports and exercise instructions from 2D videos. Mixed reality instructions have great potential for physical training, but existing works require substantial time and cost to create these 3D experiences. Video2MR overcomes this limitation by transforming arbitrary instructional videos available online into MR 3D avatars with AI-enabled motion capture (DeepMotion). Then, it automatically enhances the avatar motion through the following augmentation techniques: 1) contrasting and highlighting differences between the user and avatar postures, 2) visualizing key trajectories and movements of specific body parts, 3) manipulation of time and speed using body motion, and 4) spatially repositioning avatars for different perspectives. Developed on Hololens 2 and Azure Kinect, we showcase various use cases, including yoga, dancing, soccer, tennis, and other physical exercises. The study results confirm that Video2MR provides more engaging and playful learning experiences, compared to existing 2D video instructions.2025KIKeiichi Ihara et al.Full-Body Interaction & Embodied InputMixed Reality WorkspacesBiosensors & Physiological MonitoringIUI
Text-to-SQL Domain Adaptation via Human-LLM Collaborative Data AnnotationText-to-SQL models, which parse natural language (NL) questions to executable SQL queries, are increasingly adopted in real-world applications. However, deploying such models in the real world often requires adapting them to the highly specialized database schemas used in specific applications. We observe that the performance of existing text-to-SQL models drops dramatically when applied to a new schema, primarily due to the lack of domain-specific data for fine-tuning. Furthermore, this lack of data for the new schema also hinders our ability to effectively evaluate the model's performance in the new domain. Nevertheless, it is expensive to continuously obtain text-to-SQL data for an evolving schema in most real-world applications. To bridge this gap, we propose SQLsynth, a human-in-the-loop text-to-SQL data annotation system. SQLsynth streamlines the creation of high-quality text-to-SQL datasets through collaboration between humans and a large language model in a structured workflow. A within-subject user study comparing SQLsynth to manual annotation and ChatGPT reveals that SQLsynth significantly accelerates text-to-SQL data annotation, reduces cognitive load, and produces datasets that are more accurate, natural, and diverse. Our code is available at https://github.com/adobe/nl_sql_analyzer.2025YTYuan Tian et al.Human-LLM CollaborationAutoML InterfacesIUI
From Following to Understanding: Investigating the Role of Reflective Prompts in AR-Guided Tasks to Promote User UnderstandingAugmented Reality (AR) is a promising medium for guiding users through tasks, yet its impact on fostering deeper task understanding remains underexplored. This paper investigates the impact of reflective prompts---strategic questions that encourage users to challenge assumptions, connect actions to outcomes, and consider hypothetical scenarios---on task comprehension and performance. We conducted a two-phase study: a formative survey and co-design sessions (N=9) to develop reflective prompts, followed by a within-subject evaluation (N=16) comparing AR instructions with and without these prompts in coffee-making and circuit assembly tasks. Our results show that reflective prompts significantly improved objective task understanding and resulted in more proactive information acquisition behaviors during task completion. These findings highlight the potential of incorporating reflective elements into AR instructions to foster deeper engagement and learning. Based on data from both studies, we synthesized design guidelines for integrating reflective elements into AR systems to enhance user understanding without compromising task performance.2025NZNandi Zhang et al.University of CalgaryAR Navigation & Context AwarenessPrototyping & User TestingCHI
Prompting an Embodied AI Agent: How Embodiment and Multimodal Signaling Affects Prompting BehaviourCurrent voice agents wait for a user to complete their verbal instruction before responding; yet, this is misaligned with how humans engage in everyday conversational interaction, where interlocutors use multimodal signaling (e.g. nodding, grunting, or looking at referred to objects) to ensure conversational grounding. We designed an embodied VR agent that exhibits multimodal signaling behaviors in response to situated prompts, by turning its head, or by visually highlighting objects being discussed or referred to. We explore how people prompt this agent to design and manipulate the objects in a VR scene. Through a Wizard of Oz study, we found that participants interacting with an agent that indicated its understanding of spatial and action references were able to prevent errors 30% of the time, and were more satisfied and confident in the agent's abilities. These findings underscore the importance of designing multimodal signalling communication techniques for future embodied agents.2025TZTianyi Zhang et al.Singapore Management UniversityFull-Body Interaction & Embodied InputVoice User Interface (VUI) DesignSocial & Collaborative VRCHI
Playing with Robots: Performing Arts Techniques for Designing and Understanding Robot Group MovementIn this work, we introduce a formal design approach derived from the performing arts to design robot group movement. In our first experiment, we worked with trained actors and professional performers in a participatory design approach to identify common group movement patterns. In a follow-up studio work, we identified twelve common group movement patterns, transposed them into a performance script, built a scale model to support the performance process, and evaluated the patterns with a senior actor under studio conditions. We evaluated our refined models with 20 volunteers in a user study in the third experiment. Results from our affective circumplex modelling suggest that the patterns elicit positive emotional responses from the users. Also, participants performed better than chance in identifying the motion patterns without prior training. Based on our results, we propose design guidelines for social robots’ behaviour and movement design to improve their overall comprehensibility in interaction.2025PMPhilippa Madill et al.University of Calgary, Department of Computer ScienceSocial Robot InteractionDance & Body Movement ComputingCHI
Signaling Human Intentions to Service Robots: Understanding the Use of Social Cues during In-Person ConversationsAs social service robots become commonplace, it is essential for them to effectively interpret human signals, such as verbal, gesture, and eye gaze, when people need to focus on their primary tasks to minimize interruptions and distractions. Toward such a socially acceptable Human-Robot Interaction, we conducted a study (N=24) in an AR-simulated context of a coffee chat. Participants elicited social cues to signal intentions to an anthropomorphic, zoomorphic, grounded technical, or aerial technical robot waiter when they were speakers or listeners. Our findings reveal common patterns of social cues over intentions, the effects of robot morphology on social cue position and conversational role on social cue complexity, and users' rationale in choosing social cues. We offer insights into understanding social cues concerning perceptions of robots, cognitive load, and social context. Additionally, we discuss design considerations on approaching, social cue recognition, and response strategies for future service robots.2025HLHanfang Lyu et al.Hong Kong University of Science and TechnologySocial Robot InteractionHuman-Robot Collaboration (HRC)CHI
Beyond Vacuuming: How Can We Exploit Domestic Robots’ Idle Time?We are increasingly adopting domestic robots (e.g., Roomba) that provide relief from mundane household tasks. However, these robots usually only spend little time executing their specific task and remain idle for long periods. They typically possess advanced mobility and sensing capabilities, and therefore have significant potential applications beyond their designed use. Our work explores this untapped potential of domestic robots in ubiquitous computing, focusing on how they can improve and support modern lifestyles. We conducted two studies: an online survey (n=50) to understand current usage patterns of these robots within homes and an exploratory study (n=12) with HCI and HRI experts. Our thematic analysis revealed 12 key dimensions for developing interactions with domestic robots and outlined over 100 use cases, illustrating how these robots can offer proactive assistance and provide privacy. Finally, we implemented a proof-of-concept prototype to demonstrate the feasibility of reappropriating domestic robots for diverse ubiquitous computing applications.2025YSYoshiaki Shiokawa et al.University of Bath, Department of Computer ScienceContext-Aware ComputingDomestic RobotsCHI
HoloChemie - Sustainable Fabrication of Soft Biochemical Holographic Devices for Ubiquitous SensingSustainable fabrication approaches and biomaterials are increasingly being used in HCI to fabricate interactive devices. However, the majority of the work has focused on integrating electronics. This paper takes a sustainable approach to exploring the fabrication of biochemical sensing devices. Firstly, we contribute a set of biochemical formulations for biological and environmental sensing with bio-sourced and environment-friendly substrate materials. Our formulations are based on a combination of enzymes derived from bacteria and fungi, plant extracts and commercially available chemicals to sense both liquid and gaseous analytes: glucose, lactic acid, pH levels and carbon dioxide. Our novel holographic sensing scheme allows for detecting the presence of analytes and enables quantitative estimation of the analyte levels. We present a set of application scenarios that demonstrate the versatility of our approach and discuss the sustainability aspects, its limitations, and the implications for bio-chemical systems in HCI.2024SRSutirtha Roy et al.Shape-Changing Materials & 4D PrintingSustainable HCIEcological Design & Green ComputingUIST
Augmented Physics: Creating Interactive and Embedded Physics Simulations from Static Textbook DiagramsWe introduce Augmented Physics, a machine learning-integrated authoring tool designed for creating embedded interactive physics simulations from static textbook diagrams. Leveraging recent advancements in computer vision, such as Segment Anything and Multi-modal LLMs, our web-based system enables users to semi-automatically extract diagrams from physics textbooks and generate interactive simulations based on the extracted content. These interactive diagrams are seamlessly integrated into scanned textbook pages, facilitating interactive and personalized learning experiences across various physics concepts, such as optics, circuits, and kinematics. Drawing from an elicitation study with seven physics instructors, we explore four key augmentation strategies: 1) augmented experiments, 2) animated diagrams, 3) bi-directional binding, and 4) parameter visualization. We evaluate our system through technical evaluation, a usability study (N=12), and expert interviews (N=12). Study findings suggest that our system can facilitate more engaging and personalized learning experiences in physics education.2024AGAditya Gunturu et al.Geospatial & Map VisualizationProgramming Education & Computational ThinkingSTEM Education & Science CommunicationUIST
SHAPE-IT: Exploring Text-to-Shape-Display for Generative Shape-Changing Behaviors with LLMsThis paper introduces text-to-shape-display, a novel approach to generating dynamic shape changes in pin-based shape displays through natural language commands. By leveraging large language models (LLMs) and AI-chaining, our approach allows users to author shape-changing behaviors on demand through text prompts without programming. We describe the foundational aspects necessary for such a system, including the identification of key generative elements (primitive, animation, and interaction) and design requirements to enhance user interaction, based on formative exploration and iterative design processes. Based on these insights, we develop SHAPE-IT, an LLM-based authoring tool for a 24 x 24 shape display, which translates the user's textual command into executable code and allows for quick exploration through a web-based control interface. We evaluate the effectiveness of SHAPE-IT in two ways: 1) performance evaluation and 2) user evaluation (N= 10). The study conclusions highlight the ability to facilitate rapid ideation of a wide range of shape-changing behaviors with AI. However, the findings also expose accuracy-related challenges and limitations, prompting further exploration into refining the framework for leveraging AI to better suit the unique requirements of shape-changing systems.2024WQWanli Qian et al.Electrical Muscle Stimulation (EMS)Shape-Changing Interfaces & Soft Robotic MaterialsUIST
Introducing AV-Sketch: An Immersive Participatory Design Tool for Automated Vehicle — Passenger InteractionIn the emerging automated vehicle (AV)—passenger interaction domain, there is no agreed-upon set of methods to design early concepts. Non-designers may find it challenging to brainstorm interfaces for unfamiliar technology like AVs. Therefore, we explore using an immersive virtual environment to enable expert and non-expert designers to actively participate in the design phases. We built AV-Sketch, an in-situ (on-site) simulator that allows the creation of automotive interfaces while being immersed in VR depicting diverse AV-passenger interactions. At first, we conducted a participatory design study (𝑁=15) by utilizing PICTIVE (Plastic Interface for Collaborative Technology) to conceptualize human-machine interfaces for AV passengers. The findings led to the design of AV-Sketch, which we tested in a design session (𝑁=10), assessing users’ design experiences. Overall, participants felt more engaged and confident with the in-situ experience, enabling better contextualization of design ideas in real-world scenarios, with improved spatial considerations and dynamic aspects of in-vehicle interfaces.2024AAAshratuz Zavin Asha et al.Automated Driving Interface & Takeover DesignSocial & Collaborative VRAutoUI
RealityEffects: Augmenting 3D Volumetric Videos with Object-Centric Annotation and Dynamic Visual EffectsThis paper introduces RealityEffects, a desktop authoring interface designed for editing and augmenting 3D volumetric videos with object-centric annotations and visual effects. RealityEffects enhances volumetric capture by introducing a novel method for augmenting captured physical motion with embedded, responsive visual effects, referred to as object-centric augmentation. In RealityEffects, users can interactively attach various visual effects to physical objects within the captured 3D scene, enabling these effects to dynamically move and animate in sync with the corresponding physical motion and body movements. The primary contribution of this paper is the development of a taxonomy for such object-centric augmentations, which includes annotated labels, highlighted objects, ghost effects, and trajectory visualization. This taxonomy is informed by an analysis of 120 edited videos featuring object-centric visual effects. The findings from our user study confirm that our direct manipulation techniques lower the barriers to editing and annotating volumetric captures, thereby enhancing interactive and engaging viewing experiences of 3D volumetric videos.2024JLJian Liao et al.Video Production & Editing3D Modeling & AnimationDIS