ThingMoji: User-Captured Cut-Outs For In-Stream Visual CommunicationLive streaming has become increasingly popular, driven by the desire for direct and real-time interactions between streamers and viewers. However, current text-based interactions and pre-defined emojis limit expressiveness, especially when referring to specific stream moments. We propose ThingMoji, a type of user-captured cut-outs to enhance user expression and foster more effective communication between streamers and their audience. ThingMojis are unique digital icons created by users by capturing snapshots and annotating specific areas at any point during the stream. We developed StreamThing, a live-streaming platform integrated with ThingMojis, to explore their use during object-focused live streaming contexts. In a user study with three in-the-wild deployments reveals the expressive use of ThingMojis in diverse live-streaming scenarios with rich visual contents. Our findings show that ThingMojis enable viewers to reference specific objects, express emotions, and create shared visual narratives. Streamers found ThingMojis valuable for facilitating on-the-fly communication around visual content and fostering playful interactions. The study also uncovered challenges in ThingMoji comprehension, issues for long-term uses of ThingMojis, and potential concerns regarding misuse. Based on these insights, we discussed new opportunities for supporting object-focused communication during live streaming environments.2025EHErzhen Hu et al.Online Interactions with Friends and StrangersCSCW
Thing2Reality: Enabling Spontaneous Creation of 3D Objects from 2D Content using Generative AI in XR MeetingsDuring remote communication, participants often share both digital and physical content, such as product designs, digital assets, and environments, to enhance mutual understanding. Recent advances in augmented communication have facilitated users to swiftly create and share digital 2D copies of physical objects from video feeds into a shared space. However, conventional 2D representations of digital objects limits spatial referencing in immersive environments. To address this, we propose Thing2Reality, an Extended Reality (XR) meeting platform that facilitates spontaneous discussions of both digital and physical items during remote sessions. With Thing2Reality, users can quickly materialize ideas or objects in immersive environments and share them as conditioned multiview renderings or 3D Gaussians. Thing2Reality enables users to interact with remote objects or discuss concepts in a collaborative manner. Our user studies revealed that the ability to interact with and manipulate 3D representations of objects significantly enhances the efficiency of discussions, with the potential to augment discussion of 2D artifacts.2025EHErzhen Hu et al.Social & Collaborative VRIdentity & Avatars in XRGenerative AI (Text, Image, Music, Video)UIST
DialogLab: Authoring, Simulating, and Testing Dynamic Human-AI Group ConversationsDesigning compelling multi-party conversations involving both humans and AI agents presents significant challenges, particularly in balancing scripted structure with emergent, human-like interactions. We introduce DialogLab, a prototyping toolkit for authoring, simulating, and testing hybrid human-AI dialogues. DialogLab provides a unified interface to configure conversational scenes, define agent personas, manage group structures, specify turn-taking rules, and orchestrate transitions between scripted narratives and improvisation. Crucially, DialogLab allows designers to introduce controlled deviations from the script—through configurable agents that emulate human unpredictability—to systematically probe how conversations adapt and recover. DialogLab facilitates rapid iteration and evaluation of complex, dynamic multi-party human-AI dialogues. An evaluation with both end users and domain experts demonstrates that DialogLab supports efficient iteration and structured verification, with applications in training, rehearsal, and research on social dynamics. Our findings show the value of integrating real-time, human-in-the-loop improvisation with structured scripting to support more realistic and adaptable multi-party conversation design.2025EHErzhen Hu et al.Conversational ChatbotsHuman-LLM CollaborationUIST
Your Hands Can Tell: Detecting Redirected Hand Movements in Virtual RealityIn-air hand interactions are prevalent in Virtual Reality (VR), and prior studies have shown that manipulating the visual movement of the hand to be different from the actual hand movement, i.e., hand redirection, could create a more immersive and engaging VR experience. However, this manipulation risks degrading task performance and, if maliciously applied, poses a threat to user safety. Such manipulations may arise from VR applications developed with intentional or inadvertent perceptual manipulations that yield harmful outcomes. We advocate for a user's prerogative to be informed of any such potential manipulations before application usage. To address this, our study introduces an \textit{Autoencoder}-based anomaly detection technique that leverages users' inherent hand movements to identify hand redirection, thereby preserving the integrity of application use. Our model is trained on regular (i.e., non-manipulated) hand movement patterns and employs a stochastic thresholding approach for anomaly detection. We validated our method through a technical evaluation involving 21 participants engaged in reaching tasks under manipulated and non-manipulated scenarios. The results demonstrated a high accuracy of hand redirection detection at 93.7%, with an F1-score of 93.9%.2025MAMd Aashikur Rahman Azim et al.University of Virginia, Department of Computer ScienceVibrotactile Feedback & Skin StimulationHand Gesture RecognitionFull-Body Interaction & Embodied InputCHI
Frappé: An Ultra Lightweight Mobile UI Framework for Rapid API-based Prototyping and Environmental DeploymentQR codes have been used as an inexpensive means to connect users to digital platforms such as websites and mobile applications. However, despite their ubiquity, QR codes are limited in purpose and can only redirect users to the URL contained within it, thereby making their use heavily network dependent which can be unsuitable for use in ephemeral scenarios and areas with limited connectivity. In this paper, we introduce Frappé, a framework capable of deploying ultra lightweight UIs to mobile devices directly through QR codes, without requiring any network connectivity. This is achieved by decomposing the UI into metadata and storing it inside the QR code, while offloading the UI functionality to API calls. We also introduce enFrappé, a WYSIWYG tool for building Frappé UIs. We demonstrate the lightweight nature of our framework through a technical evaluation, whereas the usability of our UI builder tool is demonstrated through a user study.2023ARAdil Rahman et al.Crowdsourcing Task Design & Quality ControlUbiquitous ComputingMobileHCI
UnifiedSense: Enabling Without-Device Gesture Interactions Using Over-the-shoulder Training Between Redundant Wearable SensorsWearable devices allow quick and convenient interactions for controlling mobile computers. However, these interactions are often device-dependent, and users cannot control devices in a way they are familiar with if they do not wear the same wearable device. This paper proposes a new method, UnifiedSense, to enable device-dependent gestures even when the device that detects such gestures is missing by utilizing sensors on other wearable devices. UnifiedSense achieves this without explicit gesture training for different devices, by training its recognition model while users naturally perform gestures. The recognizer uses the gestures detected on the primary device (i.e., a device that reliably detects gestures) as labels for training samples and collects sensor data from all other available devices on the user. We conducted a technical evaluation with data collected from 15 participants with four types of wearable devices. It showed that UnifiedSense could correctly recognize 5 gestures (5 gestures × 5 configurations) with an accuracy of 90.9% (SD = 1.9%) without the primary device present.2023MAMd Aashikur Rahman Azim et al.Haptic WearablesHand Gesture RecognitionHuman Pose & Activity RecognitionMobileHCI
OpenMic: Utilizing Proxemic Metaphors for Conversational Floor Transitions in Multiparty Video MeetingsTurn-taking is one of the biggest interactivity challenges in multiparty remote meetings. One contributing factor is that current videoconferencing tools lack support for proxemic cues; i.e., spatial cues that humans use to enact their social relations and intentions. While more recent tools provide support for proxemic metaphors, they often focus on approach and leave-taking rather than turn-taking. In this paper, we present OpenMic, a videoconferencing system that utilizes proxemic metaphors for conversational floor management by providing 1) a Virtual Floor that serves as a fixed-feature space for users to be aware of others' intention to talk, and 2) Malleable Mirrors, which are video and screen feeds that can be continuously moved and resized for conversational floor transitions. Our exploratory user study found that these system features can aid the conversational flow in multiparty video meetings. With this work, we show potential for embedding proxemic metaphors to support conversational floor management in videoconferencing systems.2023EHErzhen Hu et al.University of VirginiaRemote Work Tools & ExperienceKnowledge Management & Team AwarenessCHI
ThingShare: Ad-Hoc Digital Copies of Physical Objects for Sharing Things in Video MeetingsIn video meetings, individuals may wish to share various physical objects with remote participants, such as physical documents, design prototypes, and personal belongings. However, our formative study discovered that this poses several challenges, including difficulties in referencing a remote user's physical objects, the limited visibility of the object, and the friction of properly framing and orienting an object to the camera. To address these challenges, we propose ThingShare, a video-conferencing system designed to facilitate the sharing of physical objects during remote meetings. With ThingShare, users can quickly create digital copies of physical objects in the video feeds, which can then be magnified on a separate panel for focused viewing, overlaid on the user’s video feed for sharing in context, and stored in the object drawer for reviews. Our user study demonstrated that ThingShare made initiating object-centric conversations more efficient and provided a more stable and comprehensive view of shared objects.2023EHErzhen Hu et al.University of VirginiaRemote Work Tools & ExperienceDistributed Team CollaborationCHI
Take My Hand: Automated Hand-Based Spatial Guidance for the Visually ImpairedTasks that involve locating objects and then moving hands to those specific locations, such as using touchscreens or grabbing objects on a desk, are challenging for the visually impaired. Over the years, audio guidance and haptic feedback have been a staple in hand navigation based assistive technologies. However, these methods require the user to interpret the generated directional cues and then manually perform the hand motions. In this paper, we present automated hand-based spatial guidance to bridge the gap between guidance and execution, allowing visually impaired users to move their hands between two points automatically, without any manual effort. We implement this concept through FingerRover, an on-finger miniature robot that carries the user's finger to target points. We demonstrate the potential applications that can benefit from automated hand-based spatial guidance. Our user study shows the potential of our technique in improving the interaction capabilities of people with visual impairments.2023ARAdil Rahman et al.University of VirginiaVibrotactile Feedback & Skin StimulationElectrical Muscle Stimulation (EMS)Haptic WearablesCHI
FluidMeet: Enabling Frictionless Transitions Between In-Group, Between-Group, and Private Conversations During Virtual Breakout MeetingsPeople often form small conversation groups during physical gatherings to have ad-hoc and informal conversations. As these groups are loosely defined, others can often overhear and join the conversation. However, current video-conferencing tools only allow for strict boundaries between small conversation groups, inhibiting fluid group formations and between-group conversations. This isolates small-group conversations from others and leads to inefficient transitions between conversations. We present FluidMeet, a virtual breakout meeting system that employs flexible conversation boundaries and cross-group conversation visualizations to enable fluid conversation group formations and ad-hoc, informal conversations. FluidMeet enables out-group members to overhear group conversations while allowing conversation groups to control their shared level of context. Users within conversation groups can also quickly switch between in-group and private conversations. A study of FluidMeet showed that it encouraged users to break group boundaries, made them feel less isolated in group conversations, and facilitated communication across different groups.2022EHErzhen Hu et al.University of VirginiaCollaborative Learning & Peer TeachingRemote Work Tools & ExperienceDistributed Team CollaborationCHI
DeepTake: Prediction of Driver Takeover Behavior using Multimodal DataAutomated vehicles promise a future where drivers can engage in non-driving tasks without hands on the steering wheels for a prolonged period. Nevertheless, automated vehicles may still need to occasionally hand the control back to drivers due to technology limitations and legal requirements. While some systems determine the need for driver takeover using driver context and road condition to initiate a takeover request, studies show that the driver may not react to it. We present DeepTake, a novel deep neural network-based framework that predicts multiple aspects of takeover behavior to ensure that the driver is able to safely take over the control when engaged in non-driving tasks. Using features from vehicle data, driver biometrics, and subjective measurements, DeepTake predicts the driver's intention, time, and quality of takeover. We evaluate DeepTake performance using multiple evaluation metrics. Results show that DeepTake reliably predicts the takeover intention, time, and quality, with an accuracy of 96%, 93%, and 83%, respectively. Results also indicate that DeepTake outperforms previous state-of-the-art methods on predicting driver takeover time and quality. Our findings have implications for the algorithm development of driver monitoring and state detection.2021EPErfan Pakdamanian et al.University of VirginiaAutomated Driving Interface & Takeover DesignCHI
MagTouch: Robust Finger Identification for a Smartwatch Using a Magnet Ring and a Built-in MagnetometerCompleting tasks on smartwatches often requires multiple gestures due to the small size of the touchscreens and the lack of sufficient number of touch controls that are easily accessible with a finger. We propose to increase the number of functions that can be triggered with the touch gesture by enabling a smartwatch to identify which finger is being used. We developed MagTouch, a method that uses a magnetometer embedded in an off-the-shelf smartwatch. It measures the magnetic field of a magnet fixed to a ring worn on the middle finger. By combining the measured magnetic field and the touch location on the screen, MagTouch recognizes which finger is being used. The tests demonstrated that MagTouch can differentiate among the three fingers used to make contacts at a success rate of 95.03%.2020KPKeunwoo Park et al.Korea Advanced Institute of Science and TechnologyHand Gesture RecognitionSmartwatches & Fitness BandsCHI