N-ary Gaussian Model Modeling Pointing Uncertainty Across Task Scenarios Using an Automated Multi-Gaussian Modeling PipelineThis paper presents an N-ary Gaussian Model for predicting endpoint distributions in pointing tasks across task scenarios. Built on the foundational principles of the Ternary Gaussian model series, our model framework allows researchers to define parameter constraints and automatically refine model combinations, eliminating the need for predefined equations based on data analysis. We utilize the Bayesian Information Criterion (BIC) for model selection, ensuring simplicity while maintaining predictive accuracy. We conducted a comparative analysis against published baselines across 7 diverse datasets, covering 1D, 2D, and 3D tasks, different input modalities, different display devices, and time-constrained scenarios, demonstrating the robustness and generalization of the N-ary Gaussian Model. The N-ary Gaussion model offers an automated solution for modeling pointing uncertainty, and also incorporates cross output device, input modality, and temporal constraint factors into spatial pointing uncertainty modeling for the first time.2026HZHao Zhang et al.Chinese Academy of SciencesTouchscreen Usability & Performance Modeling (Fitts' Law)Touch Target Selection & PointingComputational Methods in HCICHI
Continuous Measurement Methods for Transient Physiological Discomfort in VR LocomotionMotion sickness, in addition to its persistent long-term effects, also exhibits short-term effects characterized as transient physiological discomfort, which changes rapidly with variations in locomotion. However, such discomforts are challenging to assess using current subjective scales and objective physiological measurements. To tackle this issue, this paper suggests continuous measurement methods designed specifically for evaluating transient physiological discomfort during VR locomotion. Through a user-elicitation study, three preferred measurement methods—'squeezing ball', 'sliding thumb', and 'rubbing thigh'—were identified. These techniques were then evaluated for reliability, validity, attention, presence, and workload, with 'sliding thumb' identified as the most effective option. The paper expands traditional measurement methods to capture users' physiological experiences in VR interactions, offering practical choices for researchers in this field along with an in-depth discussion of design considerations, detailed implementation guidelines, and potential ways to optimize the VR experiences utilizing the measurement data.2026TLTianren Luo et al.Institute of SoftwareMotion Sickness & Passenger ExperienceImmersion & Presence ResearchCHI
ElectroGrasp: Electrotactile Aids for Visually Impaired Individuals in Anticipatory Planning and Control of GraspGrasping objects typically relies on visual input to pre-shape the hand and plan movement trajectories, a process often disrupted in visually impaired (VI) individuals. ElectroGrasp is a wearable electro-tactile system that delivers anticipatory proprioceptive and tactile information through three complementary modalities: Grasping Orientation, Size, and Shape. This system dynamically conveys spatial features-thereby enhancing anticipatory grasp planning and control through tactile perception. Three experiments were conducted to evaluate ElectroGrasp. The first examined tactile pattern discriminability, size perception thresholds, and the reliability of orientation encoding. The second assessed learning time with ElectroGrasp and its effectiveness in supporting spatial representation, demonstrating accurate spatial perception of objects from electrotactile input. The third compared grasp aperture under audio versus electrotactile cues, revealing that ElectroGrasp reduced hand overshoot and regrasp corrections. Overall, the results demonstrate that ElectroGrasp provides efficient tactile information, enables improved anticipatory grasp planning comparable to visual cues, and offers a novel assistive solution for VI users.2026HZHechuan Zhang et al.Institute of Software, Chinese Academy of SciencesVibrotactile Feedback & Skin StimulationHaptic WearablesVisual Impairment Technologies (Screen Readers, Tactile Graphics, Braille)CHI
ShakeSense: An Electrotactile System to Simulate Shaking a Container with Fluid ContentsShaking a cup of wine or other fluids in virtual environments is engaging but has been limited by challenges in delivering real-time haptic feedback for liquid collisions. ShakeSense is a haptic rendering system that integrates electrotactile stimulation with physics-based simulation to deliver immersive feedback for liquid dynamics in handheld containers. It employs a high-density electrode array to deliver dynamic tactile sensations, conveying friction and pressure changes on the user's fingerpad. A dedicated end-to-end pipeline computes fingerpad forces from liquid-container-finger interactions, ensuring feedback aligns with natural fluid movement. Two studies evaluated ShakeSense’s performance and user perception. Study 1 showed that electrotactile patterns were distinguishable across directions, and synchronizing container movement with stimulation enhanced perceived force changes. Study 2 demonstrated that ShakeSense effectively simulated liquid motion, capturing multidimensional, coordinated interactions, and outperformed conventional Center-of-Mass approaches. Overall, ShakeSense provides clear, fine-grained tactile feedback for fluid interactions.2026ZHZhenxuan He et al.Institute of software, Chinese Academy of SciencesMid-Air Haptics (Ultrasonic)Shape-Changing Interfaces & Soft Robotic MaterialsImmersion & Presence ResearchCHI
SketchGPT: A Sketch-based Multimodal Interface for Application-Agnostic LLM InteractionHuman interaction with large language models (LLMs) is typically confined to text or image interfaces. Sketches offer a powerful medium for articulating creative ideas and user intentions, yet their potential remains underexplored. We propose SketchGPT, a novel interaction paradigm that integrates sketch and speech input directly over the system interface, facilitating open-ended, context-aware communication with LLMs. By leveraging the complementary strengths of multimodal inputs, expressions are enriched with semantic scope while maintaining efficiency. Interpreting user intentions across diverse contexts and modalities remains a key challenge. To address this, we developed a prototype based on a multi-agent framework that infers user intentions within context and generates executable context-sensitive and toolkit-aware feedback. Using Chain-of-Thought techniques for temporal and semantic alignment, the system understands multimodal intentions and performs operations following human-in-the-loop confirmation to ensure reliability. User studies demonstrate that SketchGPT significantly outperforms unimodal manipulation approaches, offering more intuitive and effective means to interact with LLMs.2025ZHZeyuan Huang et al.Voice User Interface (VUI) DesignHuman-LLM CollaborationUIST
PANDA: Parkinson's Assistance and Notification Driving AidParkinson's Disease (PD) significantly impacts driving abilities, often leading to early driving cessation or accidents due to reduced motor control and increasing reaction times. To diminish the impact of these symptoms, we developed PANDA (Parkinson's Assistance and Notification Driving Aid), a multi-modality real-time alert system designed to monitor driving patterns continuously and provide immediate alerts for irregular driving behaviors, enhancing driver safety of individuals with PD. The system was developed through a participatory design process with 9 people with PD and 13 non-PD individuals using a driving simulator, which allowed us to identify critical design characteristics and collect detailed data on driving behavior. A user study involving individuals with PD evaluated the effectiveness of PANDA, exploring optimal strategies for delivering alerts and ensuring they are timely and helpful. Our findings demonstrate that PANDA has the potential to enhance the driving safety of individuals with PD, offering a valuable tool for maintaining independence and confidence behind the wheel.2025TWTianyang Wen et al.Institude of Software, Chinese Academy of SciencesIn-Vehicle Haptic, Audio & Multimodal FeedbackMotor Impairment Assistive Input TechnologiesPrototyping & User TestingCHI
Exploring the Remapping Impact of Spatial Head-hand Relations in Immersive TelesurgeryThe action remapping between the user and the avatar creates significant perceptual and behavioral challenges. Recently, in addition to virtual environments, remapping has also given rise to new applications—immersive teleoperated robots. This paper selects immersive telesurgery, a representative scenario, as an opportunity for research, exploring the generalized effects of remapping. In such a scenario, the operator can observe through the robot's camera and use their hands to control the robotic arms, as if they were the robot. However, common remapping of spatial head-hand relations—due to camera adjustments and robotic arm switching—creates significant visual-proprioceptive conflicts and physical limitations. To explore this, we simulated a telesurgery system with 6 head-camera and 12 hand-robotic-arm remapping conditions, assessing non-surgeon participants across four surgical tasks: navigation, location, cutting, and bimanual coordination. The study examines spatial perception bias, interaction deviation, workload, and task completion time. Our findings reveal how different remapping targets, attributes, intensities, and situations affect performance, contributing to the understanding of perception mechanisms and offering insights for optimizing operations or systems.2025TLTianren Luo et al.Institute of Software, Chinese Academy of Sciences; College of Computer Science and Technology, University of Chinese Academy of SciencesTeleoperated DrivingHuman-Robot Collaboration (HRC)CHI
Emotionally Challenging Games Can Satisfy Older Adults' Psychological Needs: From Empirical Study to Design GuidelinesOlder adults often struggle to meet their psychological needs due to retirement and living alone. Recent studies suggest that games featuring emotional challenge (EC) can help fulfill basic psychological needs such as autonomy, competence, and relatedness by facilitating emotional exploration. However, it remains unclear whether older adults can benefit from EC games, whether they find this genre enjoyable, and how these games should be designed to better meet their needs. This work explores older adults’ experiences and perceptions of playing EC games through two studies. The first study involved playing Detroit: Become Human, revealing that older adults derived multifaceted psychological experiences from playing the game. The second study involved a custom-designed game scenario tailored to older adults, demonstrating that meaningful choices significantly influenced autonomy need satisfaction. Based on these findings, we offer five design guidelines for developing EC games that satisfy psychological needs of older adults.2025MZMin Zhou et al.Institute of Software, ChineseAging-Friendly Technology DesignSerious & Functional GamesCHI
Slip-Grip: An Electrotactile Method to Simulate WeightWeight perception is crucial for immersive virtual reality (VR) interactions, yet providing weight feedback remains a significant research challenge. We introduce a novel weight simulation technique that leverages electrotactile stimulation to induce slip illusions. These slip illusions occur when users grip an object with less force than a predefined threshold, allowing the device to modulate the grip force and encourage a tighter grip. In our approach, heavier virtual weights correspond to higher required grip forces. We conducted a series of user experiments to validate our technique, confirming that it effectively induces slip illusions. We also investigated the relationship between electrotactile sensations and grip force, and changes in force, demonstrating that this association enhances the weight perception experience. Lastly, we explored the mapping between grip force and perceived weight, observing strong linearity within participants but notable variability between individuals.2025HLHongnan Lin et al.Institute of Software, Chinese Academy of SciencesForce Feedback & Pseudo-Haptic WeightElectrical Muscle Stimulation (EMS)CHI
RemapVR: An Immersive Authoring Tool for Rapid Prototyping of Remapped Interaction in VRRemapping techniques in VR such as repositioning, redirection, and resizing have been extensively studied. Still, interaction designers rarely have the opportunity to use them due to high technical and knowledge barriers. In the paper, we extract common features of 24 existing remapping techniques and develop a high-fidelity immersive authoring tool, namely RemapVR, for rapidly building and experiencing prototypes of remapped space properties in VR that are unperceivable or acceptable to users. RemapVR provides designers with a series of functions for editing remappings and visualizing spatial property changes, mapping relationships between real and virtual worlds, sensory conflicts, etc. Designers can quickly build existing remappings via templates, and author new remappings by interactively recording spatial relations between input trajectory in real world and output trajectory in virtual world. User studies showed that the designs of RemapVR can effectively improve designers' authoring experience and efficiency, and support designers to author remapping prototypes that meet scene requirements and provide good user experience.2025TLTianren Luo et al.Institute of Software, Chinese Academy of Sciences; College of Computer Science and Technology, University of Chinese Academy of SciencesMixed Reality WorkspacesPrototyping & User TestingCHI
TutorCraftEase: Enhancing Pedagogical Question Creation with Large Language ModelsPedagogical questions are crucial for fostering student engagement and learning. In daily teaching, teachers pose hundreds of questions to assess understanding, enhance learning outcomes, and facilitate the transfer of theory-rich content. However, even experienced teachers often struggle to generate a large volume of effective pedagogical questions. To address this, we introduce TutorCraftEase, an interactive generation system that leverages large language models (LLMs) to assist teachers in creating pedagogical questions. TutorCraftEase enables the rapid generation of questions at varying difficulty levels with a single click, while also allowing for manual review and refinement. In a comparative user study with 39 participants, we evaluated TutorCraftEase against a traditional manual authoring tool and a basic LLM tool. The results show that TutorCraftEase can generate pedagogical questions comparable in quality to those created by experienced teachers, while significantly reducing their workload and time.2025WKWenhui Kang et al.University of Chinese Academy of Sciences; Institute of Software, Chinese Academy of Sciences, Beijing Key Laboratory of Human-Computer InteractionHuman-LLM CollaborationOnline Learning & MOOC PlatformsIntelligent Tutoring Systems & Learning AnalyticsCHI
Exploring the Effects of Sensory Conflicts on Cognitive Fatigue in VR RemappingsVirtual reality (VR) is found to present significant cognitive challenges due to its immersive nature and frequent sensory conflicts. This study systematically investigates the impact of sensory conflicts induced by VR remapping techniques on cognitive fatigue, and unveils their correlation. We utilized three remapping methods (haptic repositioning, head-turning redirection, and giant resizing) to create different types of sensory conflicts, and measured perceptual thresholds to induce various intensities of the conflicts. Through experiments involving cognitive tasks along with subjective and physiological measures, we found that all three remapping methods influenced the onset and severity of cognitive fatigue, with visual-vestibular conflict having the greatest impact. Interestingly, visual-experiential/memory conflict showed a mitigating effect on cognitive fatigue, emphasizing the role of novel sensory experiences. This study contributes to a deeper understanding of cognitive fatigue under sensory conflicts and provides insights for designing VR experiences that align better with human perceptual and cognitive capabilities.2024TLTianren Luo et al.Eye Tracking & Gaze InteractionImmersion & Presence ResearchUIST
Understanding the Effects of Restraining Finger Coactivation in Mid-Air Typing: from a Neuromechanical PerspectiveTyping in mid-air is often perceived as intuitive yet presents challenges due to finger coactivation, a neuromechanical phenomenon that involves involuntary finger movements stemming from the lack of physical constraints. Previous studies were used to examine and address the impacts of finger coactivation using algorithmic approaches. Alternatively, this paper explores the neuromechanical effects of finger coactivation on mid-air typing, aiming to deepen our understanding and provide valuable insights to improve these interactions. We utilized a wearable device that restrains finger coactivation as a prop to conduct two mid-air studies, including a rapid finger-tapping task and a ten-finger typing task. The results revealed that restraining coactivation not only reduced mispresses, which is a classic coactivated error always considered as harm caused by coactivation. Unexpectedly, the reduction of motor control errors and spelling errors, thinking as non-coactivated errors, also be observed. Additionally, the study evaluated the neural resources involved in motor execution using functional Near Infrared Spectroscopy (fNIRS), which tracked cortical arousal during mid-air typing. The findings demonstrated decreased activation in the primary motor cortex of the left hemisphere when coactivation was restrained, suggesting a diminished motor execution load. This reduction suggests that a portion of neural resources is conserved, which also potentially aligns with perceived lower mental workload and decreased frustration levels.2024HZHechuan Zhang et al.Full-Body Interaction & Embodied InputUIST
WieldingCanvas: Interactive Sketch Canvases for Freehand Drawing in VRSketching in Virtual Reality (VR) is challenging mainly due to the absence of physical surface support and virtual depth perception cues, which induce high cognitive and sensorimotor load. This paper presents WieldingCanvas, an interactive VR sketching platform that integrates canvas manipulations to draw lines and curves in 3D. Informed by real-life examples of two-handed creative activities. WieldingCanvas interprets users' spatial gestures to move, swing, rotate, transform, or fold a virtual canvas, whereby users simply draw primitive strokes on the canvas, which are turned into finer and more sophisticated shapes via the manipulation of the canvas. We evaluated the capability and user experience of WieldingCanvas with three studies where participants were asked to sketch target shapes. A set of freehand sketches of high aesthetic qualities were created, and the results demonstrated that WieldingCanvas can assist users with creating 3D sketches.2024XTXiaohui Tan et al.Capital Normal UniversityMixed Reality Workspaces3D Modeling & AnimationInteractive Narrative & Immersive StorytellingCHI
TacTex: A Textile Interface with Seamlessly-Integrated Electrodes for High-Resolution electrotactile StimulationThis paper presents TacTex, a textile-based interface that provides high-resolution haptic feedback and touch-tracking capabilities. TacTex utilizes electrotactile stimulation, which has traditionally posed challenges due to limitations in textile electrode density and quantity. TacTex overcomes these challenges by employing a multi-layer woven structure that separates conductive weft and warp electrodes with non-conductive yarns. The driving system for TacTex includes a power supply, sensing board, and switch boards to enable spatial and temporal control of electrical stimuli on the textile, while simultaneously monitoring voltage changes. TacTex can stimulate a wide range of haptic effects, including static and dynamic patterns and different sensation qualities, with a resolution of $512 \times 512$ and \textcolor{black}{based on linear electrodes spaced as closely as 2mm}. We evaluate the performance of the interface with user studies and demonstrate the potential applications of TacTex interfaces in everyday textiles for adding haptic feedback.2024HLHongnan Lin et al.Institute of Software, Chinese Academy of SciencesVibrotactile Feedback & Skin StimulationShape-Changing Interfaces & Soft Robotic MaterialsElectronic Textiles (E-textiles)CHI
Exploring Experience Gaps Between Active and Passive Users During Multi-user Locomotion in VRMulti-user locomotion in VR has grown increasingly common, posing numerous challenges. A key factor contributing to these challenges is the gaps in experience between active and passive users during co-locomotion. Yet, there remains a limited understanding of how and to what extent these experiential gaps manifest in diverse multi-user co-locomotion scenarios. This paper systematically explores the gaps in physiological and psychological experience indicators between active and passive users across various locomotion situations. Such situations include when active users walk, fly by joystick, or teleport, and passive users stand still or look around. We also assess the impact of factors such as sub-locomotion type, speed/teleport-interval, motion sickness susceptibility, etc. Accordingly, we delineate acceptability disparities between active and passive users, offering insights into leveraging notable experimental findings to mitigate discomfort during co-locomotion through avoidance or intervention.2024TLTianren Luo et al.Institute of Software, College of Computer Science and TechnologySocial & Collaborative VRImmersion & Presence ResearchCHI
MathAssist: A Handwritten Mathematical Expression Autocomplete TechniqueWriting and editing mathematical expressions with complicated structures in computer system is difficult and time-consuming. To address this, we proposed MathAssist, a mathematical expression autocomplete technique that recommends full formulas in real-time based on the user's input strokes. Our technique identifies user's input purpose by matching the structure of the current user input to the structure of formulas in a database. To facilitate such process, we propose a novel tree-based formalization to represent formula. In comparison to a mathematical expression recognition algorithm (SRD) and a commercial MicroSoft Ink Equation (InkEqu), our approach outperformed both of them on task completion time (reduced by 37.14% and 37.58%) and accuracy (32.78% and 10.55% higher). We also discuss our findings in using autocomplete to assist formula editing.2024WKWenhui Kang et al.Generative AI (Text, Image, Music, Video)Programming Education & Computational ThinkingPrototyping & User TestingIUI
Exploring Locomotion Methods with Upright Redirected Views for VR Users in Reclining & Lying PositionsUsing VR in reclining & lying positions is getting common for users, but upward views caused by posture have to be redirected to be parallel to the ground as when users are standing. This affects users' locomotion performances in VR due to potential physical restrictions, and the visual-vestibular-proprioceptive conflict. This paper is among the first to investigate the suited locomotion methods and how reclining & lying positions and redirection affect them in such conditions. A user-elicitation study was carried out to construct a set of locomotion methods based on users' preferences when they were in different reclining & lying positions. A second study developed user-preferred 'tapping' and 'chair rotating' gestures, by evaluating their performances at various body reclining angles, we measured the general impacts of posture and redirection. The results showed that these methods worked effectively, but exposed some shortcomings, and users performed worst at 45-degree reclining angles. Finally, four upgraded methods were designed and verified to improve the locomotion performances.2023TLTianren Luo et al.Full-Body Interaction & Embodied InputImmersion & Presence ResearchIdentity & Avatars in XRUIST
ChallengeDetect: Investigating the Potential of Detecting In-Game Challenge Experience from Physiological MeasuresChallenge is the core element of digital games. The wide spectrum of physical, cognitive, and emotional challenge experiences provided by modern digital games can be evaluated subjectively using a questionnaire, the CORGIS, which allows for a post hoc evaluation of the overall experience that occurred during game play. Measuring this experience dynamically and objectively, however, would allow for a more holistic view of the moment-to-moment experiences of players. This study, therefore, explored the potential of detecting perceived challenge from physiological signals. For this, we collected physiological responses from 32 players who engaged in three typical game scenarios. Using perceived challenge ratings from players and extracted physiological features, we applied multiple machine learning methods and metrics to detect challenge experiences. Results show that most methods achieved a detection accuracy of around 80%. We discuss in-game challenge perception, challenge-related physiological indicators and AI-supported challenge detection to inform future work on challenge evaluation.2023XPXiaolan Peng et al.Institute of software,Chinese Academy of SciencesAI-Assisted Decision-Making & AutomationGame UX & Player BehaviorCHI
HapTag: A Compact Actuator for Rendering Push-Button Tactility on Soft SurfacesAs touch interactions become ubiquitous in the field of human computer interactions, it is critical to enrich haptic feedback to improve efficiency, accuracy and immersive experiences. This paper presents HapTag, a thin and flexible actuator to support integration of push button tactile renderings to daily soft surfaces. Specifically, HapTag works under the principle of hydraulically amplified electroactive actuator (HASEL) while being optimized by embedding a pressure sensing layer, and being activated with dedicated voltage appliance in response to users' input actions, resulting in fast response time, controllable and expressive push-button tactile rendering capabilities. HapTag is in compact formfactor, and can be attached, integrated, or embedded on various soft surfaces like cloth, leather and rubber. Three common push button tactile patterns were adopted and implemented with HapTag. We validated the feasibility and expressiveness of HapTag by demonstrating a series of innovative applications under different circumstances.2022YCYanjun Chen et al.Vibrotactile Feedback & Skin StimulationHaptic WearablesShape-Changing Interfaces & Soft Robotic MaterialsUIST