UltraEdit: In-Situ Design Environment for Ultrasound HaptizationWe present UltraEdit, an in-situ design environment that enables users to directly edit ultrasound haptic sensations in VR using barehand interactions. UltraEdit represents haptic sensations as tangible objects, called blobs, which users can modify or apply to 3D objects through intuitive hand gestures. Users can create, edit, copy, and apply blobs, allowing them to work with multiple haptic sensations to design virtual objects with diverse tactile feedback. Additionally, UltraEdit allows users to create spatial and temporal haptic patterns using time blobs and spatial drawing features. Users can draw custom-shaped haptic patterns directly on their hands, adapting to comprehensive design scenarios. To evaluate UltraEdit, we conducted an exploratory user study assessing its usability, effectiveness, and ease of learning. We also compared its performance to an existing desktop-based haptic editing tool. Participants found UltraEdit intuitive to learn, enjoyable to use, and effective for adding haptic feedback to virtual objects.2025RNRichard Huynh Noeske et al.Mid-Air Haptics (Ultrasonic)Shape-Changing Interfaces & Soft Robotic MaterialsHand Gesture RecognitionUIST
HeatFlow: A Thermal-Tactile Display for Dynamic 2D Thermal MovementsWe introduce HeatFlow, a thermal display capable of generating dynamic thermal movements on a 2D surface by integrating thermal and tactile sensations. Leveraging a perception-driven approach, we combine vibration-induced thermal referral and apparent tactile motion to create the illusion of moving thermal flows. The system features a 2D array of nine tactile actuators paired with a single nearby thermal actuator. To validate our approach, we conducted three user studies. The first study determined the optimal parameters for perceivable 2D thermal flows by examining different array sizes and motion speeds under varying temperature conditions. The second study focused on optimizing our algorithm to produce smooth and continuous curved thermal flows. Comparative evaluations against two prior tactile motion techniques demonstrated that our algorithm outperforms existing methods in generating realistic thermal motion. Finally, we integrated HeatFlow into virtual reality (VR) environments, assessing its feasibility in six interactive scenarios across three body sites—the arm, palm, and face. Our findings highlight the potential of HeatFlow to enhance immersive VR experiences by providing realistic thermal feedback across multiple body locations. Additionally, we present HeatFlow as a design tool—an application that enables users to create custom thermal flows by adjusting key parameters such as motion speed, intensity, and temperature. This work lays the foundation for more advanced thermal interfaces in interactive systems.2025YSYatharth Singhal et al.Mid-Air Haptics (Ultrasonic)Shape-Changing Interfaces & Soft Robotic MaterialsFull-Body Interaction & Embodied InputUIST
AROMA: Mixed-Initiative AI Assistance for Non-Visual Cooking by Grounding Multimodal Information Between Reality and VideosVideos offer rich audiovisual information that can support people in performing activities of daily living (ADLs), but they remain largely inaccessible to blind or low-vision (BLV) individuals. In cooking, BLV people often rely on non-visual cues---such as touch, taste, and smell---to navigate their environment, making it difficult to follow the predominantly audiovisual instructions found in video recipes. To address this problem, we introduce AROMA, an AI system that provides timely responses to the user based on real-time, context-aware assistance by integrating non-visual cues perceived by the user, a wearable camera feed, and video recipe content. AROMA uses a mixed-initiative approach: it responds to user requests while also proactively monitoring the video stream to offer timely alerts and guidance. This collaborative design leverages the complementary strengths of the user and AI system to align the physical environment with the video recipe, helping the user interpret their current cooking state and make sense of the steps. We evaluated AROMA through a study with eight BLV participants and offered insights for designing interactive AI systems to support BLV individuals in performing ADLs.2025ZNZheng Ning et al.Conversational ChatbotsVisual Impairment Technologies (Screen Readers, Tactile Graphics, Braille)Context-Aware ComputingUIST
VRSight: An AI-Driven Scene Description System to Improve Virtual Reality Accessibility for Blind PeopleVirtual Reality (VR) is inaccessible to blind people. While research has investigated many techniques to enhance VR accessibility, they require additional developer effort to integrate. As such, most mainstream VR apps remain inaccessible as the industry de-prioritizes accessibility. We present VRSight, an end-to-end system that recognizes VR scenes post hoc through a set of AI models (e.g., object detection, depth estimation, LLM-based atmosphere interpretation) and generates tone-based, spatial audio feedback, empowering blind users to interact in VR without developer intervention. To enable virtual element detection, we further contribute DISCOVR, a VR dataset consisting of 30 virtual object classes from 17 social VR apps, substituting real-world datasets that remain not applicable to VR contexts. Nine participants used VRSight to explore an off-the-shelf VR app (Rec Room), demonstrating its effectiveness in facilitating social tasks like avatar awareness and available seat identification.2025DKDaniel Killough et al.Social & Collaborative VRExplainable AI (XAI)Visual Impairment Technologies (Screen Readers, Tactile Graphics, Braille)UIST
PropType: Everyday Props as Typing Surfaces in Augmented RealityWe introduce PropType, an interactive interface that transforms everyday objects into typing surfaces within an Augmented Reality (AR) environment. Users can interact with nearby props, such as cups, water bottles, boxes, and various other objects, utilizing them as on-the-go keyboards. To develop PropType, we conducted three studies. The first study involved observing users to understand how they naturally engage with prop surfaces for typing. The second study assessed the reachability and efficiency of touch input across four props with different sizes and shapes. Based on these insights, we designed customized keyboard layouts for each prop. In the third study, we evaluated typing performance using PropType, achieving an average typing speed of up to 26.1 words per minute (WPM) with 2.2% corrected error rate (CER) and 1.1% uncorrected error rate (UER). Finally, we present a PropType editing tool that allows users to customize keyboard layouts and visual effects for prop-based typing.2025HGHyunjae Gil et al.The University of Texas at Dallas, Department of Computer ScienceHand Gesture RecognitionAR Navigation & Context AwarenessCHI
Let It Snow: Designing Snowfall Experience in VRWang等人设计并评估VR中的雪景体验，通过多感官反馈技术创造沉浸式自然现象模拟，提升用户真实感。2024HWHaokun Wang et al.Immersion & Presence ResearchUbiComp
Thermal In Motion: Designing Thermal Flow Illusions with Tactile and Thermal InteractionThis study presents a novel method for creating moving thermal sensations by integrating the thermal referral illusion with tactile motion. Conducted through three experiments on human forearms, the first experiment examined the impact of temperature and thermal actuator placement on perceived thermal motion, finding the clearest perception with a centrally positioned actuator under both hot and cold conditions. The second experiment identified the speed thresholds of perceived thermal motion, revealing a wider detectable range in hot conditions (1.8 cm/s to 9.5cm/s) compared to cold conditions (2.4cm/s to 5.0cm/s). Finally, we integrated our approach into virtual reality (VR) to assess its feasibility through two interaction scenarios. Our results shed light on the comprehension of thermal perception and its integration with tactile cues, promising significant advancements in incorporating thermal motion into diverse thermal interfaces for immersive VR experiences.2024YSYatharth Singhal et al.Vibrotactile Feedback & Skin StimulationImmersion & Presence ResearchUIST
Fiery Hands: Designing Thermal Glove through Thermal and Tactile Integration for Virtual Object ManipulationWe present a novel approach to render thermal and tactile feedback to the palm and fingertips through thermal and tactile integration. Our approach minimizes the obstruction of the palm and inner side of the fingers and enables virtual object manipulation while providing localized and global thermal feedback. By leveraging thermal actuators positioned strategically on the outer palm and back of the fingers in interplay with tactile actuators, our approach exploits thermal referral and tactile masking phenomena. Through a series of user studies, we validate the perception of localized thermal sensations across the palm and fingers, showcasing the ability to generate diverse thermal patterns. Furthermore, we demonstrate the efficacy of our approach in VR applications, replicating diverse thermal interactions with virtual objects. This work represents significant progress in thermal interactions within VR, offering enhanced sensory immersion at an optimal energy cost.2024HWHaokun Wang et al.Vibrotactile Feedback & Skin StimulationEye Tracking & Gaze InteractionImmersion & Presence ResearchUIST
CookAR: Affordance Augmentations in Wearable AR to Support Kitchen Tool Interactions for People with Low VisionCooking is a central activity of daily living, supporting independence as well as mental and physical health. However, prior work has highlighted key barriers for people with low vision (LV) to cook, particularly around safely interacting with tools, such as sharp knives or hot pans. Drawing on recent advancements in computer vision (CV), we present CookAR, a head-mounted AR system with real-time object affordance augmentations to support safe and efficient interactions with kitchen tools. To design and implement CookAR, we collected and annotated the first egocentric dataset of kitchen tool affordances, fine-tuned an affordance segmentation model, and developed an AR system with a stereo camera to generate visual augmentations. To validate CookAR, we conducted a technical evaluation of our fine-tuned model as well as a qualitative lab study with 10 LV participants for suitable augmentation design. Our technical evaluation demonstrates that our model outperforms the baseline on our tool affordance dataset, while our user study indicates a preference for affordance augmentations over the traditional whole object augmentations.2024JLJaewook Lee et al.AR Navigation & Context AwarenessVisual Impairment Technologies (Screen Readers, Tactile Graphics, Braille)Deaf & Hard-of-Hearing Support (Captions, Sign Language, Vibration)UIST
MIMOSA: Human-AI Co-Creation of Computational Spatial Audio Effects on VideosSpatial audio offers more immersive video consumption experiences to viewers; however, creating and editing spatial audio often expensive and requires specialized equipment and skills, posing a high barrier for amateur video creators. We present MIMOSA, a human-AI co-creation tool that enables amateur users to computationally generate and manipulate spatial audio effects. For a video with only monaural or stereo audio, MIMOSA automatically grounds each sound source to the corresponding sounding object in the visual scene and enables users to further validate and fix the errors in the locations of sounding objects. Users can also augment the spatial audio effect by flexibly manipulating the sounding source positions and creatively customizing the audio effect. The design of MIMOSA exemplifies a human-AI collaboration approach that, instead of utilizing state-of-art end-to-end "black-box" ML models, uses a multistep pipeline that aligns its interpretable intermediate results with the user’s workflow. A lab user study with 15 participants demonstrates MIMOSA’s usability, usefulness, expressiveness, and capability in creating immersive spatial audio effects in collaboration with users.2024ZNZheng Ning et al.Generative AI (Text, Image, Music, Video)Music Composition & Sound Design ToolsCreative Collaboration & Feedback SystemsC&C
SPICA: Interactive Video Content Exploration through Augmented Audio Descriptions for Blind or Low-Vision ViewersBlind or Low-Vision (BLV) users often rely on audio descriptions (AD) to access video content. However, conventional static ADs can leave out detailed information in videos, impose a high mental load, neglect the diverse needs and preferences of BLV users, and lack immersion. To tackle these challenges, we introduce SPICA, an AI-powered system that enables BLV users to interactively explore video content. Informed by prior empirical studies on BLV video consumption, SPICA offers novel interactive mechanisms for supporting temporal navigation of frame captions and spatial exploration of objects within key frames. Leveraging an audio-visual machine learning pipeline, SPICA augments existing ADs by adding interactivity, spatial sound effects, and individual object descriptions without requiring additional human annotation. Through a user study with 14 BLV participants, we evaluated the usability and usefulness of SPICA and explored user behaviors, preferences, and mental models when interacting with augmented ADs.2024ZNZheng Ning et al.University of Notre DameVisual Impairment Technologies (Screen Readers, Tactile Graphics, Braille)CHI
Thermal Masking: When the Illusion Takes Over the RealThis paper reports on a thermal illusion called thermal masking. Thermal masking is a phenomenon induced by thermal referral to completely mask the original thermal sensation, providing thermal sensation only at the tactile site. Three experiments are conducted using thermal and vibrotactile actuators to investigate the nature of thermal masking on human arms. The first experiment investigates the effects of different temperatures on masking. The results show a higher percentage of thermal masking occurs in warm than hot or cold conditions. The second experiment examines how far the thermal masking can be perceived. The results show that masking can reach up to 24 cm from the thermal site. The third experiment explores the interaction space by placing the tactile actuators on the opposite side of the thermal actuator. The results confirm that thermal masking can reach the other side of the arm, and the performance was higher in warm conditions.2024HWHaokun Wang et al.University of Texas at DallasVibrotactile Feedback & Skin StimulationForce Feedback & Pseudo-Haptic WeightCHI
PEANUT: A Human-AI Collaborative Tool for Annotating Audio-Visual DataAudio-visual learning seeks to enhance the computer’s multi-modal perception leveraging the correlation between the auditory and visual modalities. Despite their many useful downstream tasks, such as video retrieval, AR/VR, and accessibility, the performance and adoption of existing audio-visual models have been impeded by the availability of high quality datasets. Annotating audio-visual datasets is laborious, expensive, and time consuming. To address this challenge, we designed and developed an efficient audio visual annotation tool called Peanut. Peanut’s human-AI collaborative pipeline separates the multi-modal task into two single-modal tasks, and utilizes state-of-the-art object detection and sound-tagging models to reduce the annotators’ effort to process each frame and the number of manually-annotated frames needed. A within-subject user study with 20 participants found that Peanut can significantly accelerate the audio-visual data annotation process while maintaining high annotation accuracy.2023ZZZheng Zhang et al.Conversational ChatbotsHuman-LLM CollaborationRecommender System UXUIST
Improving Finger Stroke Recognition Rate for Eyes-Free Mid-Air Typing in VRWe examine mid-air typing data collected from touch typists to evaluate the features and classification models for recognizing finger stroke. A large number of finger movement traces have been collected using finger motion capture systems, labeled into individual finger strokes, and classified into several key features. We test finger kinematic features, including 3D position, velocity, acceleration, and temporal features, including previous fingers and keys. Based on this analysis, we assess the performance of various classifiers, including Naive Bayes, Random Forest, Support Vector Machines, and Deep Neural Networks, in terms of the accuracy for correctly classifying the keystroke. We finally incorporate a linguistic heuristic to explore the effectiveness of the character prediction model and improve the total accuracy.2022YSYatharth Singhal et al.University of Texas at DallasEye Tracking & Gaze InteractionUbiquitous ComputingCHI
Designing Transparency Cues in Online News Platforms to Promote Trust: Journalists' & Consumers' Perspectives As news organizations embrace transparency practices on their websites to distinguish themselves from those spreading misinformation, HCI designers have the opportunity to help them effectively utilize the ideals of transparency to build trust. How can we utilize transparency to promote trust in news? We examine this question through a qualitative lens by interviewing journalists and news consumers---the two stakeholders in a news system. We designed a scenario to demonstrate transparency features using two fundamental news attributes that convey the trustworthiness of a news article: source and message. In the interviews, our news consumers expressed the idea that news transparency could be best shown by providing indicators of objectivity in two areas (news selection and framing) and by providing indicators of evidence in four areas (presence of source materials, anonymous sourcing, verification, and corrections upon erroneous reporting). While our journalists agreed with news consumers' suggestions of using evidence indicators, they also suggested additional transparency indicators in areas such as the news reporting process and personal/organizational conflicts of interest. Prompted by our scenario, participants offered new design considerations for building trustworthy news platforms, such as designing for easy comprehension, presenting appropriate details in news articles (e.g., showing the number and nature of corrections made to an article), and comparing attributes across news organizations to highlight diverging practices. Comparing the responses from our two stakeholder groups reveals conflicting suggestions with trade-offs between them. Our study has implications for HCI designers in building trustworthy news systems.2021MBMd Momen Bhuiyan et al.Misinformation, Conspiracies, and ManipulationsCSCW
Anchoring Bias Affects Mental Model Formation and User Reliance in Explainable AI SystemsEXplainable Artificial Intelligence (XAI) approaches are used to bring transparency to machine learning and artificial intelligence models, and hence, improve the decision-making process for their end-users. While these methods aim to improve human understanding and thier mental models, cognitive biases can still influence a user's mental model and decision-making in ways that system designers do not anticipate. This paper presents research on cognitive biases due to ordering effects in intelligent systems. We conducted a controlled user study to understand how the order of observing system weaknesses and strengths can affect the user's mental model, task performance, and reliance on the intelligent system, and we investigate the role of explanations in addressing this bias. Using an explainable video activity recognition tool in the cooking domain, we asked participants to verify whether a set of kitchen policies are being followed, with each policy focusing on a weakness or a strength. We controlled the order of the policies and the presence of explanations to test our hypotheses. Our main finding shows that those who observed system strengths early-on were more prone to automation bias and made significantly more errors due to positive first impressions of the system, while they built a more accurate mental model of the system competencies. On the other hand, those who encountered weaknesses earlier made significantly fewer errors since they tended to rely more on themselves, while they also underestimated model competencies due to having a more negative first impression of the model. Our work presents strong findings that aims to make intelligent system designers aware of such biases when designing such tools.2021MNMahsan Nourani et al.Explainable AI (XAI)Context-Aware ComputingIUI
Challenges of Designing Consent: Consent Mechanics in Video Games as Models for Interactive User AgencyThis paper argues for a conceptual framework that treats user consent in interactive technologies as a design challenge necessitating careful, culturally-informed consideration. We draw on recent work in HCI as well as queer and feminist theory that understands consent as rooted in negotiating agency in order to frame our exploration of unique difficulties and potential solutions to meaningful opportunities for user consent in the design of computational technologies. Through a critical analysis of three video games that offer different models of consent—each of which communicates different values through its design—we introduce the concept of consent mechanics. Consent mechanics describe designed interactions that allow players to consent to or opt out of in-game experiences, often those related to sexuality or intimacy. Here, we approach video games as windows onto design considerations surrounding interactive technologies more broadly, suggesting crucial questions and tactics for how to design user agency ethically into computational systems.2020JNJosef Nguyen et al.The University of Texas at DallasAgent Personality & AnthropomorphismGender & Race Issues in HCITechnology Ethics & Critical HCICHI
Towards Human-Guided Machine LearningAutomated Machine Learning (AutoML) systems are emerging that automatically search for possible solutions from a large space of possible kinds of models. Although fully automated machine learning is appropriate for many applications, users often have knowledge that supplements and constraints the available data and solutions. This paper proposes human-guided machine learning (HGML) as a hybrid approach where a user interacts with an AutoML system and tasks it to explore different problem settings that reflect the user’s knowledge about the data available. We present: 1) a task analysis of HGML that shows the tasks that a user would want to carry out, 2) a characterization of two scientific publications, one in neuroscience and one in political science, in terms of how the authors would search for solutions using an AutoML system, 3) requirements for HGML based on those characterizations, and 4) an assessment of existing AutoML systems in terms of those requirements.2019YGYolanda Gil et al.Human-LLM CollaborationAI-Assisted Decision-Making & AutomationAutoML InterfacesIUI