Gazeify Then Voiceify: Physical Object Referencing Through Gaze and Voice Interaction with Displayless Smart GlassesSmart glasses enhance interactions with the environment by using head-mounted cameras to observe the user’s viewpoint , but lack the visual feedback used for common interactions. We introduce "Gazeify then Voiceify", a multimodal approach allowing object selection via gaze and voice using displayless smart glasses. Users can select a physical object with their gaze, and the system generates a digital mask and a voice description of the object's semantics. Users can further correct errors through free-form conversation. To demonstrate our approach, we develop an interactive system by integrating advanced object segmentation and detection with a visual-language model. User studies reveal that participants achieve correct gaze selection in 53% of the task trials and use voice disambiguation to correct 58% remaining errors. Participants also rated the system as likable, useful and easy to use.2026ZZZheng Zhang et al.University of Notre DameEye Tracking & Gaze InteractionVoice User Interface (VUI) DesignContext-Aware ComputingIUI
Gesturing Toward Abstraction: Multimodal Convention Formation in Collaborative Physical TasksA quintessential feature of human intelligence is the ability to create ad hoc conventions over time to achieve shared goals efficiently. We investigate how communication strategies evolve through repeated collaboration as people coordinate on shared procedural abstractions. To this end, we conducted an online unimodal study (n = 98) using natural language to probe abstraction hierarchies. In a follow-up lab study (n = 40), we examined how multimodal communication (speech and gestures) changed during physical collaboration. Pairs used augmented reality to isolate their partner’s hand and voice; one participant viewed a 3D virtual tower and sent instructions to the other, who built the physical tower. Participants became faster and more accurate by establishing linguistic and gestural abstractions and using cross-modal redundancy to emphasize key changes from previous interactions. Based on these findings, we extend probabilistic models of convention formation to multimodal settings, capturing shifts in modality preferences. Our findings and model provide building blocks for designing convention-aware intelligent agents situated in the physical world.2026KMKiyosu Maeda et al.Princeton UniversityFull-Body Interaction & Embodied InputEye Tracking & Gaze InteractionAR Navigation & Context AwarenessCHI
Investigating How Physical Surfaces Can Serve as Common-Region Cues for Perceptual Grouping of Virtual Elements in Augmented RealityPerceptual grouping enables people to organize elements into units according to intrinsic (e.g., proximity) and extrinsic (e.g., common region) principles. However, the role of physical surfaces as extrinsic grouping cues for virtual elements in Augmented Reality (AR) remains unclear. To provide a deeper understanding, we conducted two within-subject studies. The first study (N = 24) using repetition discrimination tasks revealed that surfaces can be common-region cues in 3D, with their influence depending on their distance to target objects along the viewing direction. Building on these findings, the second study (N = 24) employed both objective and subjective measures to capture the interaction between proximity and common-region cues in AR. Results indicate that competing cues reduce group clarity. They also enable us to distill people's strategies for improving the clarity by leveraging their physical and virtual environments. Finally, we propose design recommendations for future AR systems in assisted grouping tasks.2026XYXuanhui Yang et al.The Hong Kong University of Science and TechnologyAR Navigation & Context AwarenessImmersion & Presence ResearchPrototyping & User TestingCHI
Understanding Parents’ Desires in Moderating Children’s Interactions with GenAI Chatbots through LLM-Generated ProbesThis paper studies how parents want to moderate children’s interactions with Generative AI Chatbots, with the goal of informing the design of future GenAI parental control tools. We first used an LLM to generate synthetic Child--GenAI Chatbot interaction scenarios and worked with four parents to validate their realism. From this dataset, we carefully selected 12 diverse examples that evoked varying levels of concern and were rated the most realistic. Each example included a prompt and GenAI Chatbot response. We presented these to parents (N=24) and asked whether they found them concerning, why, and how they would prefer to modify the responses and be informed. Our findings reveal three key insights: (1) parents express concern about interactions that current GenAI Chatbot parental controls neglect; (2) parents want fine-grained transparency and moderation at the conversation level; and (3) parents need personalized controls that adapt to their desired strategies and children's ages.2026JDJohn Driscoll et al.University of California San DiegoConversational ChatbotsMental Health Technology for YouthChildren's AI Literacy & Data LiteracyCHI
Design and Evaluation of a Photorealistic AI Virtual Peer in Elementary Collaborative ClassroomIn elementary education, students struggle to articulate uncertainties, limiting diverse perspectives in classroom discussions, particularly in small schools where limited participants constrain collaborative learning. This study designed and evaluated ``Saya,'' a photorealistic AI virtual peer functioning as an additional student. We implemented five teacher-controlled speech acts (expand, probe, summarize, lighten, and incorrect answer) through dynamic classroom dialogue generation using GPT-4o-mini. Field studies in Japanese elementary schools (large class: 27 students, small class: 2 students) demonstrated that Saya integration increased the proportion of student speaking time by 1.28 times and 2.07 times respectively, with 95.6% and 100% of students expressing desire for future Saya-integrated lessons. Teachers reported enhanced student concentration and listening behaviors, noting that interactions with Saya prompted students to reconstruct their own understanding of the learning material. This research provides new insights into design principles for collaborative learning agents in elementary education settings, effective implementation scenarios based on class size, and the future potential of AI-enhanced collaborative learning.2026STSatomi Tokida et al.The University of TokyoProgramming Education & Computational ThinkingCollaborative Learning & Peer TeachingHuman-LLM CollaborationCHI
CoBRA: Programming Cognitive Bias in Social Agents Using Classic Social Science ExperimentsThis paper introduces CoBRA, a novel toolkit for systematically specifying agent behavior in LLM-based social simulation. We found that conventional approaches that specify agent behavior through implicit natural-language descriptions often do not yield consistent behavior across models, and the resulting behavior does not capture the nuances of the descriptions. In contrast, CoBRA introduces a model-agnostic way to control agent behavior that lets researchers explicitly specify desired nuances and obtain consistent behavior across models. At the heart of CoBRA is a novel closed-loop system primitive with two components:(1) Cognitive Bias Index that measures the demonstrated cognitive bias of a social agent, by quantifying the agent’s reactions in a set of validated classic social science experiments; (2) Behavioral Regulation Engine that aligns the agent’s behavior to exhibit controlled cognitive bias. Through CoBRA, we show how to operationalize validated social science knowledge (i.e., classical experiments) as reusable “gym” environments for AI—an approach that may generalize to richer social and affective simulations beyond bias alone.2026XLXuan Liu et al.University of California San DiegoHuman-LLM CollaborationExplainable AI (XAI)Brain-Computer Interface (BCI) & NeurofeedbackCHI
PrivacyAkinator: Articulating Key Privacy Design Decisions by Answering LLM-Generated Multiple-choice QuestionsNIST's Privacy Risk Assessment Methodology (PRAM) provides a structured framework for privacy experts to assess privacy risks. However, its complexity and reliance on expert knowledge make it difficult for novice developers to use effectively. This paper explores methods to lower these barriers. We first performed an observational study with 12 participants using PRAM in real-world scenarios, and found that novice developers struggled most with articulating privacy-related design decisions. We then developed PrivacyAkinator, an interactive tool that helps developers articulate key privacy decisions by answering LLM-generated multiple-choice questions. PrivacyAkinator introduces three innovations: a universal privacy representation that abstracts privacy-related design decisions into data flows and stakeholder interactions; a domain-aware design space mined from 10K privacy-related news articles; and a dynamic question-generation workflow to prioritize relevant questions. Our user study with 24 participants suggests that developers using PrivacyAkinator identified 47% more key decisions in 73% less time compared to PRAM.2026QLQiyu Li et al.University of California San DiegoExplainable AI (XAI)Privacy by Design & User ControlPrivacy Perception & Decision-MakingCHI
Can GenAI Move from Individual Use to Collaborative Work? Experiences, Challenges, and Opportunities of Coordinating GenAI into Collaborative NewsworkGenerative AI (GenAI) is reshaping work, but adoption remains largely individual and experimental rather than coordinated into collaborative work. Whether GenAI can move from individual use to collaborative work is a critical question for future organizations. Journalism offers a compelling site to examine this shift: individual journalists have already been disrupted by GenAI tools; yet newswork is inherently collaborative relying on shared norms and coordinated workflows. We conducted 27 interviews with newsroom managers, editors and front-line journalists in China. We found that journalists frequently used GenAI to support daily tasks, but value alignment was safeguarded mainly through individual discretion. At the organizational level, GenAI use remained disconnected from team workflows, hindered by structural barriers and cultural reluctance to share practices. These findings underscore the gap between individual and collaborative work, pointing to the need to account for organizational structures, cultural norms, and workflow when coordinating GenAI for collaborative work.2026QXQing Xiao et al.Carnegie Mellon UniversityGenerative AI (Text, Image, Music, Video)Human-LLM CollaborationAI-Assisted Decision-Making & AutomationCHI
VizCrit: Exploring Strategies for Displaying Computational Feedback in a Visual Design ToolVisual design instructors often provide multi-modal feedback, mixing annotations with text. Prior theory emphasizes the importance of actionable feedback, where “actionability” lies on a spectrum—from surfacing relevant design concepts to suggesting concrete fixes. How might creativity tools implement annotations that support such feedback, and how does the actionability of feedback impact novices’ process-related behaviors, perceptions of creativity, learning of design principles, and overall outcomes? We introduce VizCrit, a system for providing computational feedback that supports the actionability spectrum, realized through algorithmic issue detection and visual annotation generation. In a between-subjects study (N=36), novices revised a design under one of three conditions: textbook-based, awareness-centered, or solution-centered feedback. We found that solution-centered feedback led to fewer design issues and higher self-perceived creativity compared with textbook-based feedback, although expert ratings on creativity showed no significant differences. We discuss the implications for AI in Creativity Support Tools, including the potential of calibrating feedback actionability to help novices balance productivity with learning, growth, and developing design awareness.2026MLMingyi Li et al.Northeastern UniversityGenerative AI (Text, Image, Music, Video)Creative Collaboration & Feedback SystemsCHI
Tidynote: Always-Clear Notebook AuthoringRecent work identified clarity as one of the top quality attributes that notebook users value, but notebooks lack support for maintaining clarity throughout the exploratory phases of the notebook authoring workflow. We propose always-clear notebook authoring that supports both clarity and exploration, and present a Jupyter implementation called Tidynote. The key to Tidynote is three-fold: (1) a scratchpad sidebar to facilitate exploration, (2) cells movable between the notebook and the scratchpad to maintain organization, and (3) linear execution with state forks to clarify program state. An exploratory study (N=13) of open-ended data analysis tasks shows that Tidynote features holistically promote clarity throughout a notebook's lifecycle, support realistic notebook tasks, and enable novel strategies for notebook clarity. These results suggest that Tidynote supports maintaining clarity throughout the entirety of notebook authoring.2026RHRuanqianqian (Lisa) Huang et al.UC San DiegoCollaborative Writing ToolsPrototyping & User TestingUser Research Methods (Interviews, Surveys, Observation)CHI
A Design Space for Live Music AgentsLive music provides a uniquely rich setting for studying creativity and interaction due to its spontaneous nature. The pursuit of live music agents---intelligent systems supporting real-time music performance and interaction---has captivated researchers across HCI, AI, and computer music for decades, and recent advancements in AI suggest unprecedented opportunities to evolve their design. However, the interdisciplinary nature of music has led to fragmented development across research communities, hindering effective communication and collaborative progress. In this work, we bring together perspectives from these diverse fields to map the current landscape of live music agents. Based on our analysis of 184 systems across both academic literature and video, we develop a comprehensive design space that categorizes dimensions spanning usage contexts, interactions, technologies, and ecosystems. By highlighting trends and gaps in live music agents, our design space offers researchers, designers, and musicians a structured lens to understand existing systems and shape future directions in real-time human-AI music co-creation. We release our annotated systems as a living artifact at https://live-music-agents.github.io.2026YKYewon Kim et al.Carnegie Mellon UniversityMusic Composition & Sound Design ToolsGenerative AI (Text, Image, Music, Video)Creative Collaboration & Feedback SystemsCHI
Orca: Browsing at Scale Through User-Driven and AI-Facilitated Orchestration Across Malleable WebpagesWeb-based activities span multiple webpages. However, conventional browsers with stacks of tabs cannot support operating and synthesizing large volumes of information across pages. While recent AI systems enable fully automated web browsing and information synthesis, they often diminish user agency and hinder contextual understanding. We explore how AI could instead augment user interactions with content across webpages and mitigate cognitive and manual efforts. Through literature on information tasks and web browsing challenges, and an iterative design process, we present novel interactions with our prototype web browser, Orca. Leveraging AI, Orca supports user-driven exploration, operation, organization, and synthesis of web content at scale. To enable browsing at scale, webpages are treated as malleable materials that humans and AI can collaboratively manipulate and compose into a malleable, dynamic, and browser-level workspace. Our evaluation revealed an increased "appetite" for information foraging, enhanced control, and more flexible sensemaking across a broader web information landscape.2026PJPeiling Jiang et al.University of California San DiegoGenerative AI (Text, Image, Music, Video)AI-Assisted Decision-Making & AutomationExploratory Search & Information SeekingCHI
Belidor: A Specification Language for Operationalizing Structural Analogies Between User InterfacesWe present Belidor, a text notation that describes the structure underlying user interfaces (UIs). Belidor’s relational model emphasizes how structures, such as the temporal order of text messages, cut across an interactive system’s conceptual model, user-facing presentation, and interactive behavior. We demonstrate Belidor’s expressive power with a gallery of examples spanning GUIs (eg. messaging, video editors), screen readers, and hardware devices. Belidor serves as an effective representation for structural analogies between user interfaces (eg. between calendars and video-editors). In contrast, prior work relied on visual UI representations and therefore prioritized visual style transfer. In three case studies, we show how Belidor can reveal analogies, help transfer ideas between user interfaces, and describe design patterns as analogies We discuss the implications of representing the structure of interactive systems for designers and developers, and envision how Belidor might support ``structural design moves'' for interface designers.2026MBMatthew Beaudouin-Lafon et al.University of California San DiegoParticipatory DesignPrototyping & User TestingComputational Methods in HCICHI
"Bespoke Bots'': Diverse Instructor Needs for Customizing Generative AI Classroom ChatbotsInstructors are increasingly experimenting with AI chatbots for classroom support. To investigate how instructors adapt chatbots to their own contexts, we first analyzed existing resources that provide prompts for educational purposes. We identified ten common categories of customization, such as persona, guardrails, and personalization. We then conducted interviews with ten university STEM instructors and asked them to card-sort the categories into priorities. We found that instructors consistently prioritized the ability to customize chatbot behavior to align with course materials and pedagogical strategies and de-prioritized customizing persona/tone. However, their prioritization of other categories varied significantly by course size, discipline, and teaching style, even across courses taught by the same individual, highlighting that no single design can meet all contexts. These findings suggest that modular AI chatbots may provide a promising path forward. We offer design implications for educational developers building the next generation of customizable classroom AI systems.2026IHIrene Hou et al.University of California, San DiegoHuman-LLM CollaborationIntelligent Tutoring Systems & Learning AnalyticsConversational ChatbotsCHI
Deception at Scale: Deceptive Designs in 1K LLM-Generated E-Commerce ComponentsRecent work has shown that front-end code generated by Large Language Models (LLMs) can embed deceptive designs. To assess the magnitude of this problem, identify the factors that influence deceptive design production, and test strategies for reducing deceptive designs, we carried out two studies which generated and analyzed 1,296 LLM-generated web components, along with a design rationale for each. The first study tested four LLMs for 15 common ecommerce components. Overall 55.8% of components contained at least one deceptive design, and 30.6% contained two or more. Occurence varied significantly across models, with DeepSeek-V3 producing the fewest. Interface interference emerged as the dominant strategy, using color psychology to influence actions and hiding essential information. The first study found that prompts emphasizing business interests (e.g., increasing sales) significantly increased deceptive designs, so a second study tested a variety of prompting strategies to reduce their frequency, finding a values-centered approach the most effective. Our findings highlight risks in using LLMs for coding and offer recommendations for LLM developers and providers.2026ZCZiwei Chen et al.University of California San DiegoAI Ethics, Fairness & AccountabilityDark Patterns RecognitionHuman-LLM CollaborationCHI
Beyond the Desk: Barriers and Future Opportunities for AI to Assist Scientists in Embodied Physical TasksMore scientists are now using AI, but prior studies have examined only how they use it `at the desk' for computer-based work. However, given that scientific work often happens `beyond the desk’ at lab and field sites, we conducted the first study of how \rev{scientific practitioners} use AI for embodied physical tasks. We interviewed 12 \rev{scientific practitioners doing hands-on lab and fieldwork} in domains like nuclear fusion, primate cognition, and biochemistry, and found three barriers to AI adoption in these settings: 1) experimental setups are too high-stakes to risk AI errors, 2) constrained environments make it hard to use AI, and 3) AI cannot match the tacit knowledge of humans. Participants then developed speculative designs for future AI assistants to 1) monitor task status, 2) organize lab-wide knowledge, 3) monitor scientists’ health, 4) do field scouting, 5) do hands-on chores. Our findings point toward AI as background infrastructure to support physical work rather than replacing human expertise.2026IHIrene Hou et al.University of California, San DiegoAI-Assisted Decision-Making & AutomationGenerative AI (Text, Image, Music, Video)Participatory DesignCHI
Sculpin: Direct-Manipulation Transformation of JSONMany end-user programming tasks require programmatically processing JSON, wrangling it from one format to another or building interactive applications atop it. But end-users are impeded by the indirectness and steep learning curve of textual code. We present Sculpin, a direct-manipulation environment supporting a broad range of JSON-transformation tasks. A user of Sculpin transforms JSON data step by step, recording a program in the process. Sculpin makes three design commitments to ensure directness and versatility: (1) steps are small and precise, not inferred; (2) steps are general-purpose and open to re-appropriation; (3) steps operate on JSON itself, rather than on a limited intermediate representation. To support these commitments, Sculpin introduces a mechanism of sculptable selections: the user can direct their action by guiding a selection on top of the data through small steps like generalization and hierarchical navigation. Sculpin also extends JSON with embedded interface elements like form inputs and buttons, allowing applications to be sculpted incrementally from source data. We demonstrate the breadth and directness of Sculpin in use-cases ranging from wrangling data to building applications. We evaluate Sculpin through a heuristic analysis, situating it in a broad space of programming systems and surfacing limitations such as difficulties editing preexisting programs.2025JHJoshua Horowitz et al.Knowledge Worker Tools & WorkflowsPrototyping & User TestingUIST
UltraPoser: Pushing the Limits of IMU-based Full-Body Pose Estimation with Ultrasound Sensing on Consumer WearablesFull-body motion capture using IMUs embedded in consumer wearables has the potential to enable convenient, on-the-go tracking with minimal instrumentation. However, the sparse placement of these devices on the body frame presents challenges such as limited body coverage, reduced motion feature diversity, and cumulative drift errors. This paper introduces UltraPoser, a multi-modal full-body motion capture system that integrates ultrasonic sensing with inertial measurements for improved fidelity, broader coverage, and increased reliability. UltraPoser leverages built-in microphones and speakers on commodity wearables, such as smartphones and smartwatches, to transmit and receive inaudible ultrasound signals, expanding the range of sensed body areas and providing drift-free acoustic multipath profiles. To implement UltraPoser, we systematically explore ultrasound signal designs to maximize feature quality and propose a graph-based physics-aware fusion architecture to integrate heterogeneous sensing modalities. We evaluate our approach using the UltraPoser Dataset, collected from 10 participants across diverse device placements and activity contexts. Compared to state-of-the-art IMU-only methods, UltraPoser achieves a 28.46% improvement in overall pose estimation accuracy and up to 67.28% error reduction for specific limbs without directly attached sensors.2025YLYadong Li et al.Mid-Air Haptics (Ultrasonic)Full-Body Interaction & Embodied InputUIST
Meridian: A Design Framework for Malleable Overview-Detail InterfacesOverview-detail interfaces (ODIs), which present an overview of multiple items alongside a detailed view of a selected item, are ubiquitously implemented in software interfaces. However, the current design and development pipeline lacks the infrastructure to easily support end-user customization, limiting its ability to support diverse information needs. This research envisions a development cycle for building malleable interfaces—one where designers, developers, and end-users alike can create, modify, and use the interface equally. To establish a foundation for this infrastructure, we introduce Meridian, a design framework for guiding and facilitating the creation of malleable ODIs. The framework consists of a high-level declarative specification language for ODIs as well as its tools, including a UI development package and a no-code web builder to facilitate the development and design of malleable ODIs. We demonstrate how Meridian supports designers, developers, and end-users alike in designing, implementing, and interacting with ODIs in novel ways using their respective familiar tools and platforms. Finally, we discuss technical tradeoffs, potential solutions, and opportunities for enabling malleability for interfaces by default.2025BMBryan Min et al.Interactive Data VisualizationKnowledge Worker Tools & WorkflowsUIST
Understanding the Challenges and Design Opportunities of Using Voice Assistants to Support Postpartum Mothers in BrazilThe postpartum period is a crucial time for physical and mental adjustment for a mother, which can worsen through increased demands and mental load. In Brazil, as in many Latin American countries, the unequal division of childcare responsibilities increases mothers’ risks of postpartum depression and anxiety while preventing mothers from focusing on their recovery. While today's Voice Assistants (VAs) are promising to offer hands-free, eyes-free, and on-demand support, it remains unclear how VAs can be designed to effectively support mothers and their associated tasks during the postpartum period. To address this challenge, we conducted an online survey study with $55$ Brazilian mothers to investigate how VAs support postpartum mothers and their current usage in childcare-related tasks. We identified key challenges preventing VAs from effectively supporting Brazilian mothers, including language barriers, lack of personalized information retrieval, and missing features tailored to postpartum care and early childhood needs. We then proposed a set of design considerations for how VAs could meet mothers' needs for greater adoption in Brazil.2025JSJessica de Souza et al.Intelligent Voice Assistants (Alexa, Siri, etc.)Mental Health Apps & Online Support CommunitiesReproductive & Women's HealthCUI