Exploring Collaborative GenAI Agents in Synchronous Group Settings: Eliciting Team Perceptions and Design Considerations for the Future of WorkWhile Generative Artificial Intelligence (GenAI) is finding increased adoption in workplaces, current tools are primarily designed for individual use. Prior work established the potential for these tools to enhance personal creativity and productivity towards shared goals; however, we don't know yet how to best take into account the nuances of group work and team dynamics when deploying GenAI in work settings. In this paper, we investigate the potential of collaborative GenAI agents to augment teamwork in synchronous group settings through an exploratory study that engaged 25 professionals across 6 teams in speculative design workshops and individual follow-up interviews. Our workshops included a Mixed Reality provotype to simulate embodied collaborative GenAI agents capable of actively participating in group discussions. Our findings suggest that, if designed well, collaborative GenAI agents offer valuable opportunities to enhance team problem-solving by challenging groupthink, bridging communication gaps, and reducing social friction. However, teams' willingness to integrate GenAI agents depended on its perceived fit across a number of individual, team, and organizational factors. We outline the key design tensions around agent representation, social prominence, and engagement and highlight the opportunities spatial and immersive technologies could offer to modulate GenAI influence on team outcomes and strike a balance between augmentation and agency.2025JJJanet G Johnson et al.Team Work Makes the Dream WorkCSCW
Privacy Equilibrium: Balancing Privacy Needs in Dynamic Multi-User Augmented Reality ScenariosAs augmented reality (AR) glasses become more widely used in public settings, a key challenge is meeting the privacy needs of multiple AR users and bystanders in a fine-grained manner. To enable this, we present a conceptual framework for Privacy Equilibrium--balancing user experience (UX) and privacy between all individuals in a shared space. The framework applies constrained optimization to compute AR sensing policies that grant or restrict permissions to maximize UX while minimizing privacy risks (e.g., capturing bystanders or sensitive environmental data). We instantiate this framework in a simulation and analysis toolkit to holistically evaluate optimization strategies and visualize tradeoffs between UX and privacy. Through application scenarios, we demonstrate the flexibility of our optimization approach to minimize these tradeoffs across conflicting user needs and privacy preferences. Walkthrough evaluations with AR and security & privacy researchers highlight the potential of our framework and toolkit to inform future privacy-mediating techniques for AR.2025SRShwetha Rajaram et al.AR Navigation & Context AwarenessPrivacy by Design & User ControlPrivacy Perception & Decision-MakingUIST
Exploring the Design Space of Privacy-Driven Adaptation Techniques for Future Augmented Reality InterfacesModern augmented reality (AR) devices with advanced display and sensing capabilities pose significant privacy risks to users and bystanders. While previous context-aware adaptations focused on usability and ergonomics, we explore the design space of privacy-driven adaptations that allow users to meet their dynamic needs. These techniques offer granular control over AR sensing capabilities across various AR input, output, and interaction modalities, aiming to minimize degradations to the user experience. Through an elicitation study with 10 AR researchers, we derive 62 privacy-focused adaptation techniques that preserve key AR functionalities and classify them into system-driven, user-driven, and mixed-initiative approaches to create an adaptation catalog. We also contribute a visualization tool that helps AR developers navigate the design space, validating its effectiveness in design workshops with six AR developers. Our findings indicate that the tool allowed developers to discover new techniques, evaluate tradeoffs, and make informed decisions that balance usability and privacy concerns in AR design.2025SRShwetha Rajaram et al.University of Michigan, School of InformationAR Navigation & Context AwarenessPrivacy by Design & User ControlContext-Aware ComputingCHI
XCam: Mixed-Initiative Virtual Cinematography for Live Production of Virtual Reality ExperiencesVR is often utilized for organizing virtual events such as meetings, conferences, and concerts; however, support for live production is lacking in most existing VR tools. We present XCam, a toolkit enabling mixed-initiative control over virtual camera systems---from fully manual control by users to increasingly automated, system-driven control with minimal user intervention. XCam's architectural design separates the concerns of object tracking, camera motion, and scene transition, giving more degrees of freedom to operators who can adjust the level of automation along all three dimensions. We used to conduct two studies: (1) interviews with six VR content creators probe into what aspects should and shouldn't be automated based on six applications developed with XCam; (2) three workshops with experts explore XCam's utility in live production of an interactive VR film sequence, a lecture on cinematography, and an alumni meeting in social VR. Expert feedback from our studies suggests how to balance automation and control, and the opportunities and limits of future AI-driven tools.2025MNMichael Nebeling et al.University of Michigan, School of InformationSocial & Collaborative VRVideo Production & EditingCHI
A Multimodal Approach for Targeting Error Detection in Virtual Reality Using Implicit User BehaviorAlthough the point-and-select interaction method has been shown to lead to user and system-initiated errors, it is still prevalent in VR scenarios. Current solutions to facilitate selection interactions exist, however they do not address the challenges caused by targeting inaccuracy. To reduce the effort required to target objects, we developed a model that quickly detected targeting errors after they occurred. The model used implicit multimodal user behavioral data to identify possible targeting outcomes. Using a dataset composed of 23 participants engaged in VR targeting tasks, we then trained a deep learning model to differentiate between correct and incorrect targeting events within 0.5 seconds of a selection, resulting in an AUC-ROC of 0.9. The utility of this model was then evaluated in a user study with 25 participants that identified that participants recovered from more errors and faster when assisted by the model. These results advance our understanding of targeting errors in VR and facilitate the design of future intelligent error-aware systems.2025NSNaveen Sendhilnathan et al.MetaSocial & Collaborative VRImmersion & Presence ResearchHuman-LLM CollaborationCHI
SonoHaptics: An Audio-Haptic Cursor for Gaze-Based Object Selection in XRWe introduce SonoHaptics, an audio-haptic cursor for gaze-based 3D object selection. SonoHaptics addresses challenges around providing accurate visual feedback during gaze-based selection in Extended Reality (XR), e.g., lack of world-locked displays in no- or limited-display smart glasses and visual inconsistencies. To enable users to distinguish objects without visual feedback, SonoHaptics employs the concept of cross-modal correspondence in human perception to map visual features of objects (color, size, position, material) to audio-haptic properties (pitch, amplitude, direction, timbre). We contribute data-driven models for determining cross-modal mappings of visual features to audio and haptic features, and a computational approach to automatically generate audio-haptic feedback for objects in the user's environment. SonoHaptics provides global feedback that is unique to each object in the scene, and local feedback to amplify differences between nearby objects. Our comparative evaluation shows that SonoHaptics enables accurate object identification and selection in a cluttered scene without visual feedback.2024HCHyunsung Cho et al.Mid-Air Haptics (Ultrasonic)Eye Tracking & Gaze InteractionSocial & Collaborative VRUIST
Reframe: An Augmented Reality Storyboarding Tool for Character-Driven Analysis of Security & Privacy ConcernsWhile current augmented reality (AR) authoring tools lower the technical barrier for novice AR designers, they lack explicit guidance to consider potentially harmful aspects of AR with respect to security & privacy (S&P). To address potential threats in the earliest stages of AR design, we developed Reframe, a digital storyboarding tool for designers with no formal training to analyze S&P threats. We accomplish this through a frame-based authoring approach, which captures and enhances storyboard elements that are relevant for threat modeling, and character-driven analysis tools, which personify S&P threats from an underlying threat model to provide simple abstractions for novice designers. Based on evaluations with novice AR designers and S&P experts, we find that Reframe enables designers to analyze threats and propose mitigation techniques that experts consider good quality. We discuss how Reframe can facilitate collaboration between designers and S\&P professionals and propose extensions to Reframe to incorporate additional threat models.2023SRShwetha Rajaram et al.AR Navigation & Context AwarenessPrivacy by Design & User ControlIoT Device PrivacyUIST
Eliciting Security & Privacy-Informed Sharing Techniques for Multi-User Augmented RealityThe HCI community has explored new interaction designs for collaborative AR interfaces in terms of usability and feasibility; however, security & privacy (S&P) are often not considered in the design process and left to S&P professionals. To produce interaction proposals with S&P in mind, we extend the user-driven elicitation method with a scenario-based approach that incorporates a threat model involving access control in multi-user AR. We conducted an elicitation study in two conditions, pairing AR/AR experts in one condition and AR/S&P experts in the other, to investigate the impact of each pairing. We contribute a set of expert-elicited interactions for sharing AR content enhanced with access control provisions, analyze the benefits and tradeoffs of pairing AR and S&P experts, and present recommendations for designing future multi-user AR interactions that better balance competing design goals of usability, feasibility, and S&P in collaborative AR.2023SRShwetha Rajaram et al.University of MichiganMixed Reality WorkspacesPrivacy by Design & User ControlCHI
Color-to-Depth Mappings as Depth Cues in Virtual RealityDespite significant improvements to Virtual Reality (VR) technologies, most VR displays are fixed focus and depth perception is still a key issue that limits the user experience and the interaction performance. To supplement humans' inherent depth cues (e.g., retinal blur, motion parallax), we investigate users' perceptual mappings of distance to virtual objects' appearance to generate visual cues aimed to enhance depth perception. As a first step, we explore color-to-depth mappings for virtual objects so that their appearance differs in saturation and value to reflect their distance. Through a series of controlled experiments, we elicit and analyze users' strategies of mapping a virtual object's hue, saturation, value and a combination of saturation and value to its depth. Based on the collected data, we implement a computational model that generates color-to-depth mappings fulfilling adjustable requirements on confusion probability, number of depth levels, and consistent saturation/value changing tendency. We demonstrate the effectiveness of color-to-depth mappings in a 3D sketching task, showing that compared to single-colored targets and strokes, with our mappings, the users were more confident in the accuracy without extra cognitive load and reduced the perceived depth error by 60.8%. We also implement four VR applications and demonstrate how our color cues can benefit the user experience and interaction performance in VR.2022ZLZhipeng Li et al.Immersion & Presence ResearchMedical & Scientific Data VisualizationUIST
Elements of XR Prototyping: Characterizing the Role and Use of Prototypes in Augmented and Virtual Reality DesignCurrent research in augmented, virtual, and mixed reality (XR) reveals a lack of tool support for designing and, in particular, prototyping XR applications. While recent tools research is often motivated by studying the requirements of non-technical designers and end-user developers, the perspective of industry practitioners is less well understood. In an interview study with 17 practitioners from different industry sectors working on professional XR projects, we establish the design practices in industry, from early project stages to the final product. To better understand XR design challenges, we characterize the different methods and tools used for prototyping and describe the role and use of key prototypes in the different projects. We extract common elements of XR prototyping, elaborating on the tools and materials used for prototyping and establishing different views on the notion of fidelity. Finally, we highlight key issues for future XR tools research.2022VKVeronika Krauß et al.University of Siegen, Bonn-Rhein Sieg University of Applied ScienceMixed Reality WorkspacesPrototyping & User TestingCHI
Paper Trail: An Immersive Authoring System for Augmented Reality Instructional ExperiencesPrior work has demonstrated augmented reality's benefits to education, but current tools are difficult to integrate with traditional instructional methods. We present Paper Trail, an immersive authoring system designed to explore how to enable instructors to create AR educational experiences, leaving paper at the core of the interaction and enhancing it with various forms of digital media, animations for dynamic illustrations, and clipping masks to guide learning. To inform the system design, we developed five scenarios exploring the benefits that hand-held and head-worn AR can bring to STEM instruction and developed a design space of AR interactions enhancing paper based on these scenarios and prior work. Using the example of an AR physics handout, we assessed the system's potential with PhD-level instructors and its usability with XR design experts. In an elicitation study with high-school teachers, we study how Paper Trail could be used and extended to enable flexible use cases across various domains. We discuss benefits of immersive paper for supporting diverse student needs and challenges for making effective use of AR for learning.2022SRShwetha Rajaram et al.University of MichiganAR Navigation & Context AwarenessK-12 Digital Education ToolsSTEM Education & Science CommunicationCHI
XRStudio: A Virtual Production and Live Streaming System for Immersive Instructional ExperiencesThere is increased interest in using virtual reality in education, but it often remains an isolated experience that is difficult to integrate into current instructional experiences. In this work, we adapt virtual production techniques from filmmaking to enable mixed reality capture of instructors so that they appear to be standing directly in the virtual scene. We also capitalize on the growing popularity of live streaming software for video conferencing and live production. With XRStudio, we develop a pipeline for giving lectures in VR, enabling live compositing using a variety of presets and real-time output to traditional video and more immersive formats. We present interviews with media designers experienced in film and MOOC production that informed our design. Through walkthrough demonstrations of XRStudio with instructors experienced with VR, we learn how it could be used in a variety of domains. In end-to-end evaluations with students, we analyze and compare differences of traditional video vs. more immersive lectures with XRStudio.2021MNMichael Nebeling et al.University of MichiganSocial & Collaborative VRMixed Reality WorkspacesOnline Learning & MOOC PlatformsCHI
Creating Augmented and Virtual Reality Applications: Current Practices, Challenges, and OpportunitiesAugmented Reality (AR) and Virtual Reality (VR) devices are becoming easier to access and use, but the barrier to entry for creating AR/VR applications remains high. Although the recent spike in HCI research on novel AR/VR tools is promising, we lack insights into how AR/VR creators use today's state-of-the-art authoring tools as well as the types of challenges that they face. We interviewed 21 AR/VR creators, which we grouped into hobbyists, domain experts, and professional designers. Despite having a variety of motivations and skillsets, they described similar challenges in designing and building AR/VR applications. We synthesize 8 key barriers that AR/VR creators face nowadays, starting from prototyping the initial experiences to dealing with "the many unknowns" during implementation, to facing difficulties in testing applications. Based on our analysis, we discuss the importance of considering end-user developers as a growing population of AR/VR creators, how we can build learning opportunities into AR/VR tools, and the need for building AR/VR toolchains that integrate debugging and testing.2020NANarges Ashtari et al.Simon Fraser UniversityMixed Reality WorkspacesPrototyping & User TestingCHI
MRAT: The Mixed Reality Analytics ToolkitSignificant tool support exists for the development of mixed reality (MR) applications; however, there is a lack of tools for analyzing MR experiences. We elicit requirements for future tools through interviews with 8 university research, instructional, and media teams using AR/VR in a variety of domains. While we find a common need for capturing how users perform tasks in MR, the primary differences were in terms of heuristics and metrics relevant to each project. Particularly in the early project stages, teams were uncertain about what data should, and even could, be collected with MR technologies. We designed the Mixed Reality Analytics Toolkit (MRAT) to instrument MR apps via visual editors without programming and enable rapid data collection and filtering for visualizations of MR user sessions. With MRAT, we contribute flexible interaction tracking and task definition concepts, an extensible set of heuristic techniques and metrics to measure task success, and visual inspection tools with in-situ visualizations in MR. Focusing on a multi-user, cross-device MR crisis simulation and triage training app as a case study, we then show the benefits of using MRAT, not only for user testing of MR apps, but also performance tuning throughout the design process.2020MNMichael Nebeling et al.University of Michigan Ann ArborMixed Reality WorkspacesInteractive Data VisualizationMedical & Scientific Data VisualizationCHI
XRDirector: A Role-Based Collaborative Immersive Authoring SystemImmersive authoring is an increasingly popular technique to design AR/VR scenes because design and testing can be done concurrently. Most existing systems, however, are single-user and limited to either AR or VR, thus constrained in the interaction techniques. We present XRDirector, a role-based collaborative immersive authoring system that enables designers to freely express interactions using AR and VR devices as puppets to manipulate virtual objects in 3D physical space. In XRDirector, we adapt roles known from filmmaking to structure the authoring process and help coordinate multiple designers in immersive authoring tasks. We study how novice AR/VR creators can take advantage of the roles and modes in XRDirector to prototype complex scenes with animated 3D characters, light effects, and camera movements, and also simulate interactive system behavior in a Wizard of Oz style. XRDirector's design was informed by case studies around complex 3D movie scenes and AR/VR games, as well as workshops with novice AR/VR creators. We show that XRDirector makes it easier and faster to create AR/VR scenes without the need for coding, characterize the issues in coordinating designers between AR and VR, and identify the strengths and weaknesses of each role and mode to mitigate the issues.2020MNMichael Nebeling et al.University of Michigan Ann ArborMixed Reality WorkspacesCreative Collaboration & Feedback SystemsCHI
360proto: Making Interactive Virtual Reality & Augmented Reality Prototypes from PaperWe explore 360 paper prototyping to rapidly create AR/VR prototypes from paper and bring them to life on AR/VR devices. Our approach is based on a set of emerging paper prototyping templates specifically for AR/VR. These templates resemble the key components of many AR/VR interfaces, including 2D representations of immersive environments, AR marker overlays and face masks, VR controller models and menus, and 2D screens and HUDs. To make prototyping with these templates effective, we developed 360proto, a suite of three novel physical--digital prototyping tools: (1) the 360proto Camera for capturing paper mockups of all components simply by taking a photo with a smartphone and seeing 360-degree panoramic previews on the phone or stereoscopic previews in Google Cardboard; (2) the 360proto Studio for organizing and editing captures, for composing AR/VR interfaces by layering the captures, and for making them interactive with Wizard of Oz via live video streaming; (3) the 360proto App for running and testing the interactive prototypes on AR/VR capable mobile devices and headsets. Through five student design jams with a total of 86 participants and our own design space explorations, we demonstrate that our approach with 360proto is useful to create relatively complex AR/VR applications.2019MNMichael Nebeling et al.University of Michigan3D Modeling & AnimationPrototyping & User TestingDigital Art Installations & Interactive PerformanceCHI
What is Mixed Reality?What is Mixed Reality (MR)? To revisit this question given the many recent developments, we conducted interviews with ten AR/VR experts from academia and industry, as well as a literature survey of 68 papers. We find that, while there are prominent examples, there is no universally agreed on, one-size-fits-all definition of MR. Rather, we identified six partially competing notions from the literature and experts' responses. We then started to isolate the different aspects of reality relevant for MR experiences, going beyond the primarily visual notions and extending to audio, motion, haptics, taste, and smell. We distill our findings into a conceptual framework with seven dimensions to characterize MR applications in terms of the number of environments, number of users, level of immersion, level of virtuality, degree of interaction, input, and output. Our goal with this paper is to support classification and discussion of MR applications' design and provide a better means to researchers to contextualize their work within the increasingly fragmented MR landscape.2019MSMaximilian Speicher et al.University of Michigan & C&A EuropeMixed Reality WorkspacesImmersion & Presence ResearchCHI
Redefining Natural User InterfaceThis SIG focuses on new definitions of Natural User Interface (NUI). With the adoption of wearable devices, VR & AR displays, affective computing, and voice user interface, we think it’s necessary to review our understanding and definition of NUI. This SIG aims to expand discussion and development related to NUI in two areas: first, what experience should NUIs achieve today? How can we build UIs to leverage other senses besides vision and hearing such as tactility, olfaction and gustation? Second, how can we detect, capture and compute people’s behavioral signals in a natural way and provide output accordingly? What are the current available technologies to achieve NUIs, and what new technologies should be invented to achieve it?2018LFLimin Paul Fu et al.Alibaba DAMO AcademyFull-Body Interaction & Embodied InputBrain-Computer Interface (BCI) & NeurofeedbackMixed Reality WorkspacesCHI
User-Driven Design Principles for Gesture RepresentationsMany recent studies have explored user-defined interactions for touch and gesture-based systems through end-user elicitation. While these studies have facilitated the user-end of the human-computer dialogue, the subsequent design of gesture representations to communicate gestures to the user vary in style and consistency. Our study explores how users interpret, enact, and refine gesture representations adapting techniques from recent elicitation studies. To inform our study design, we analyzed gesture representations from 30 elicitation papers and developed a taxonomy of design elements. We then conducted a partnered elicitation study with 30 participants producing 657 gesture representations accompanied by think-aloud data. We discuss design patterns and themes that emerged from our analysis, and supplement these findings with an in-depth look at users’ mental models when perceiving and enacting gesture representations. Finally, based on the results, we provide recommendations for practitioners in need of “visual language” guidelines to communicate possible user actions.2018EMErin McAweeney et al.University of Michigan, University of WashingtonHand Gesture RecognitionEye Tracking & Gaze InteractionCHI
GestureWiz: A Human-Powered Gesture Design Environment for User Interface PrototypesDesigners and researchers often rely on simple gesture recognizers like Wobbrock et al.'s $1 for rapid user interface prototypes. However, most existing recognizers are limited to a particular input modality and/or pre-trained set of gestures, and cannot be easily combined with other recognizers. In particular, creating prototypes that employ advanced touch and mid-air gestures still requires significant technical experience and programming skill. Inspired by $1's easy, cheap, and flexible design, we present the GestureWiz prototyping environment that provides designers with an integrated solution for gesture definition, conflict checking, and real-time recognition by employing human recognizers in a Wizard of Oz manner. We present a series of experiments with designers and crowds to show that GestureWiz can perform with reasonable accuracy and latency. We demonstrate advantages of GestureWiz when recreating gesture-based interfaces from the literature and conducting a study with 12 interaction designers that prototyped a multimodal interface with support for a wide range of novel gestures in about 45 minutes.2018MSMaximilian Speicher et al.University of MichiganHand Gesture RecognitionPrototyping & User TestingCHI