Smiles, Frowns, and Everything In Between: Understanding User Experiences with Emotion Tracking in Daily Life through Facial Expression RecognitionEmotion tracking is commonly practiced for mental wellbeing. In recent years, increasing attention is turned to automatic emotion tracking, including technologies based on Facial Expression Recognition (FER). However, work on FER based tracking so far has mainly focused on accuracy improvement, and little is on how it works in everyday lives. In this paper, we developed an FER-based emotion tracking web application, EmoAction, and used it as a technology probe to explore participants' experiences and perceptions of using FER-based emotion tracking in daily life. A field study with 9 participants trying it over 14 days reveals how such an emotion tracking tool enables people to gain new awareness and new perspectives of their emotions, and identifies several potential usages of such an automatic tracking tool. We then discuss the implications for future design, including providing real-time non-judgmental feedback, prioritizing triggering reflection over seeking maximum precision, and enabling social sharing with respect for privacy.2025ZYZhennan Yi et al.Human Pose & Activity RecognitionCognitive Impairment & Neurodiversity (Autism, ADHD, Dyslexia)DIS
Cracking Aegis: An Adversarial LLM-based Game for Raising Awareness of Vulnerabilities in Privacy ProtectionTraditional methods for raising awareness of privacy protection often fail to engage users or provide hands-on insights into how privacy vulnerabilities are exploited. To address this, we incorporate an adversarial mechanic in the design of the dialogue-based serious game Cracking Aegis. Leveraging LLMs to simulate natural interactions, the game challenges players to impersonate characters and extract sensitive information from an AI agent, Aegis. A user study (n=22) revealed that players employed diverse deceptive linguistic strategies, including storytelling and emotional rapport, to manipulate Aegis. After playing, players reported connecting in-game scenarios with real-world privacy vulnerabilities, such as phishing and impersonation, and expressed intentions to strengthen privacy control, such as avoiding oversharing personal information with AI systems. This work highlights the potential of LLMs to simulate complex relational interactions in serious games, while demonstrating how an adversarial strategy provides a unique perspective in designing for social good, particularly in privacy protection.2025JFJiaying Fu et al.Serious & Functional GamesPrivacy Perception & Decision-MakingDark Patterns RecognitionDIS
Exploring the Remapping Impact of Spatial Head-hand Relations in Immersive TelesurgeryThe action remapping between the user and the avatar creates significant perceptual and behavioral challenges. Recently, in addition to virtual environments, remapping has also given rise to new applications—immersive teleoperated robots. This paper selects immersive telesurgery, a representative scenario, as an opportunity for research, exploring the generalized effects of remapping. In such a scenario, the operator can observe through the robot's camera and use their hands to control the robotic arms, as if they were the robot. However, common remapping of spatial head-hand relations—due to camera adjustments and robotic arm switching—creates significant visual-proprioceptive conflicts and physical limitations. To explore this, we simulated a telesurgery system with 6 head-camera and 12 hand-robotic-arm remapping conditions, assessing non-surgeon participants across four surgical tasks: navigation, location, cutting, and bimanual coordination. The study examines spatial perception bias, interaction deviation, workload, and task completion time. Our findings reveal how different remapping targets, attributes, intensities, and situations affect performance, contributing to the understanding of perception mechanisms and offering insights for optimizing operations or systems.2025TLTianren Luo et al.Institute of Software, Chinese Academy of Sciences; College of Computer Science and Technology, University of Chinese Academy of SciencesTeleoperated DrivingHuman-Robot Collaboration (HRC)CHI
RemapVR: An Immersive Authoring Tool for Rapid Prototyping of Remapped Interaction in VRRemapping techniques in VR such as repositioning, redirection, and resizing have been extensively studied. Still, interaction designers rarely have the opportunity to use them due to high technical and knowledge barriers. In the paper, we extract common features of 24 existing remapping techniques and develop a high-fidelity immersive authoring tool, namely RemapVR, for rapidly building and experiencing prototypes of remapped space properties in VR that are unperceivable or acceptable to users. RemapVR provides designers with a series of functions for editing remappings and visualizing spatial property changes, mapping relationships between real and virtual worlds, sensory conflicts, etc. Designers can quickly build existing remappings via templates, and author new remappings by interactively recording spatial relations between input trajectory in real world and output trajectory in virtual world. User studies showed that the designs of RemapVR can effectively improve designers' authoring experience and efficiency, and support designers to author remapping prototypes that meet scene requirements and provide good user experience.2025TLTianren Luo et al.Institute of Software, Chinese Academy of Sciences; College of Computer Science and Technology, University of Chinese Academy of SciencesMixed Reality WorkspacesPrototyping & User TestingCHI
Exploring the Effects of Sensory Conflicts on Cognitive Fatigue in VR RemappingsVirtual reality (VR) is found to present significant cognitive challenges due to its immersive nature and frequent sensory conflicts. This study systematically investigates the impact of sensory conflicts induced by VR remapping techniques on cognitive fatigue, and unveils their correlation. We utilized three remapping methods (haptic repositioning, head-turning redirection, and giant resizing) to create different types of sensory conflicts, and measured perceptual thresholds to induce various intensities of the conflicts. Through experiments involving cognitive tasks along with subjective and physiological measures, we found that all three remapping methods influenced the onset and severity of cognitive fatigue, with visual-vestibular conflict having the greatest impact. Interestingly, visual-experiential/memory conflict showed a mitigating effect on cognitive fatigue, emphasizing the role of novel sensory experiences. This study contributes to a deeper understanding of cognitive fatigue under sensory conflicts and provides insights for designing VR experiences that align better with human perceptual and cognitive capabilities.2024TLTianren Luo et al.Eye Tracking & Gaze InteractionImmersion & Presence ResearchUIST
ChatScratch: An AI-Augmented System Toward Autonomous Visual Programming Learning for Children Aged 6-12As Computational Thinking (CT) continues to permeate younger age groups in K-12 education, established CT platforms such as Scratch face challenges in catering to these younger learners, particularly those in the elementary school (ages 6-12). Through formative investigation with Scratch experts, we uncover three key obstacles to children's autonomous Scratch learning: artist's block in project planning, bounded creativity in asset creation, and inadequate coding guidance during implementation. To address these barriers, we introduce ChatScratch, an AI-augmented system to facilitate autonomous programming learning for young children. ChatScratch employs structured interactive storyboards and visual cues to overcome artist's block, integrates digital drawing and advanced image generation technologies to elevate creativity, and leverages Scratch-specialized Large Language Models (LLMs) for professional coding guidance. Our study shows that, compared to Scratch, ChatScratch efficiently fosters autonomous programming learning, and contributes to the creation of high-quality, personally meaningful Scratch projects for children.2024LCLiuqing Chen et al.Zhejiang UniversityHuman-LLM CollaborationProgramming Education & Computational ThinkingK-12 Digital Education ToolsCHI
Technology-Mediated Non-pharmacological Interventions for Dementia: Needs for and Challenges in Professional, Personalized and Multi-Stakeholder Collaborative InterventionsDesigning and using technologies to support Non-Pharmacological Interventions (NPI) for People with Dementia (PwD) has drawn increasing attention in HCI, with the potential expectations of higher user engagement and positive outcomes. Yet, technologies for NPI can only be valuable if practitioners successfully incorporate them into their ongoing intervention practices beyond a limited research period. Currently, we know little about how practitioners experience and perceive these technologies in practical NPI for PwD. In this paper, we investigate this question through observations of five in-person NPI activities and interviews with 11 therapists and 5 caregivers. Our findings elaborate the practical NPI workflow process and characteristics, and practitioners’ attitudes, experiences, and perceptions to technology-mediated NPI in practice. Generally, our participants emphasized practical NPI is a complex and professional practice, needing fine-grained, personalized evaluation and planning, and the practical executing process is situated, and multi-stakeholder collaborative. Yet, existing technologies often fail to consider these specific characteristics, which leads to limitations in practical effectiveness or sustainable use. Drawing on our findings, we discuss the possible implications for designing more useful and practical NPI intervention technologies.2024YSYuling Sun et al.East China Normal UniversityElderly Care & Dementia SupportCHI
Exploring Locomotion Methods with Upright Redirected Views for VR Users in Reclining & Lying PositionsUsing VR in reclining & lying positions is getting common for users, but upward views caused by posture have to be redirected to be parallel to the ground as when users are standing. This affects users' locomotion performances in VR due to potential physical restrictions, and the visual-vestibular-proprioceptive conflict. This paper is among the first to investigate the suited locomotion methods and how reclining & lying positions and redirection affect them in such conditions. A user-elicitation study was carried out to construct a set of locomotion methods based on users' preferences when they were in different reclining & lying positions. A second study developed user-preferred 'tapping' and 'chair rotating' gestures, by evaluating their performances at various body reclining angles, we measured the general impacts of posture and redirection. The results showed that these methods worked effectively, but exposed some shortcomings, and users performed worst at 45-degree reclining angles. Finally, four upgraded methods were designed and verified to improve the locomotion performances.2023TLTianren Luo et al.Full-Body Interaction & Embodied InputImmersion & Presence ResearchIdentity & Avatars in XRUIST
From 'Wow' to 'Why': Guidelines for Creating the Opening of a Data Video with Cinematic StylesData videos are an increasingly popular storytelling form. The opening of a data video critically influences its success as the opening either attracts the audience to continue watching or bores them to abandon watching. However, little is known about how to create an attractive opening. We draw inspiration from the openings of famous films to facilitate designing data video openings. First, by analyzing over 200 films from several sources, we derived six primary cinematic opening styles adaptable to data videos. Then, we consulted eight experts from the film industry to formulate 28 guidelines. To validate the usability and effectiveness of the guidelines, we asked participants to create data video openings with and without the guidelines, which were then evaluated by experts and the general public. Results showed that the openings designed with the guidelines were perceived to be more attractive, and the guidelines were praised for clarity and inspiration.2022XXXian Xu et al.The Hong Kong University of Science and TechnologyData StorytellingVideo Production & EditingCHI
RestoreVR: Generating Embodied Knowledge and Situated Experience of Dunhuang Mural Conservation via Interactive Virtual RealityIn Dunhuang Mogao Grottoes, unique Buddhist murals of ancient China are preserved. Unfortunately, the exquisite murals are suffering from degradation. Experts have been trying to enhance public's awareness of mural protection, but there's no efficacious means to attract interest and popularize knowledge yet. In this paper, we propose RestoreVR, an interactive virtual reality (VR) system engaging users to experience Dunhuang mural restoration in a digital tour in the cave. Based on an online survey with the public and in-depth interviews with five Dunhuang experts, we derive a set of design requirements for generating embodied knowledge and situated experience in VR to bridge the gap between highly specialized experts and general audiences. Accordingly, we design RestoreVR and conduct a between-subjects user study to compare our system with traditional methods. The results suggest that RestoreVR significantly improves user experience and awareness of CH protection over existing methods.2020XFXinyi Fu et al.Tsinghua UniversitySocial & Collaborative VRMuseum & Cultural Heritage DigitizationInteractive Narrative & Immersive StorytellingCHI