SketchGPT: A Sketch-based Multimodal Interface for Application-Agnostic LLM InteractionHuman interaction with large language models (LLMs) is typically confined to text or image interfaces. Sketches offer a powerful medium for articulating creative ideas and user intentions, yet their potential remains underexplored. We propose SketchGPT, a novel interaction paradigm that integrates sketch and speech input directly over the system interface, facilitating open-ended, context-aware communication with LLMs. By leveraging the complementary strengths of multimodal inputs, expressions are enriched with semantic scope while maintaining efficiency. Interpreting user intentions across diverse contexts and modalities remains a key challenge. To address this, we developed a prototype based on a multi-agent framework that infers user intentions within context and generates executable context-sensitive and toolkit-aware feedback. Using Chain-of-Thought techniques for temporal and semantic alignment, the system understands multimodal intentions and performs operations following human-in-the-loop confirmation to ensure reliability. User studies demonstrate that SketchGPT significantly outperforms unimodal manipulation approaches, offering more intuitive and effective means to interact with LLMs.2025ZHZeyuan Huang et al.Voice User Interface (VUI) DesignHuman-LLM CollaborationUIST
Assessing Dynamic Flow Experience from EEG Signals: A Processing-based ApproachAs an interaction experience goal, the flow experience is characterized by its subjectivity and dynamism. Exploring objective methods to assess dynamic flow states is significant in enhancing user experience design, evaluation, and optimization. This study aims to model the dynamics of the flow experience and quantify its intensity using electroencephalography signals (EEG) from the perspective of the process. To achieve this, an interactive task is designed to induce dynamic changes in flow, and EEG signals from participants were recorded simultaneously, to form a flow assessment dataset. Subsequently, a frequency-aware convolutional Transformer model (FA-ConFormer) was proposed to extract dynamic features from EEG, with particular optimization for capturing complex dynamic features in the frequency domain. Experimental results demonstrate that FA-ConFormer outperforms existing methods in flow state and intensity recognition, the visualization of the flow process dynamically depicting the onset, development, peak, and decline of flow with varying intensities, which help to deepen the understanding of flow experience.2025SLJuan Liu et al.Brain-Computer Interface (BCI) & NeurofeedbackVisualization Perception & CognitionUIST
Unknown Word Detection for English as a Second Language (ESL) Learners using Gaze and Pre-trained Language ModelsEnglish as a Second Language (ESL) learners often encounter unknown words that hinder their text comprehension. Automatically detecting these words as users read can enable computing systems to provide just-in-time definitions, synonyms, or contextual explanations, thereby helping users learn vocabulary in a natural and seamless manner. This paper presents EyeLingo, a transformer-based machine learning method that predicts the probability of unknown words based on text content and eye gaze trajectory in real time with high accuracy. A 20-participant user study revealed that our method can achieve an accuracy of 97.6%, and an F1-score of 71.1%. We implemented a real-time reading assistance prototype to show the effectiveness of EyeLingo. The user study shows improvement in willingness to use and usefulness compared to baseline methods.2025JDJiexin Ding et al.Tsinghua University, Key Laboratory of Pervasive Computing, Ministry of Education, Department of Computer Science and Technology, Global Innovation Exchange (GIX) Institute; University of Washington, Paul G. Allen School of Computer Science & EngineeringHuman Pose & Activity RecognitionHuman-LLM CollaborationCHI
PANDA: Parkinson's Assistance and Notification Driving AidParkinson's Disease (PD) significantly impacts driving abilities, often leading to early driving cessation or accidents due to reduced motor control and increasing reaction times. To diminish the impact of these symptoms, we developed PANDA (Parkinson's Assistance and Notification Driving Aid), a multi-modality real-time alert system designed to monitor driving patterns continuously and provide immediate alerts for irregular driving behaviors, enhancing driver safety of individuals with PD. The system was developed through a participatory design process with 9 people with PD and 13 non-PD individuals using a driving simulator, which allowed us to identify critical design characteristics and collect detailed data on driving behavior. A user study involving individuals with PD evaluated the effectiveness of PANDA, exploring optimal strategies for delivering alerts and ensuring they are timely and helpful. Our findings demonstrate that PANDA has the potential to enhance the driving safety of individuals with PD, offering a valuable tool for maintaining independence and confidence behind the wheel.2025TWTianyang Wen et al.Institude of Software, Chinese Academy of SciencesIn-Vehicle Haptic, Audio & Multimodal FeedbackMotor Impairment Assistive Input TechnologiesPrototyping & User TestingCHI
Exploring the Remapping Impact of Spatial Head-hand Relations in Immersive TelesurgeryThe action remapping between the user and the avatar creates significant perceptual and behavioral challenges. Recently, in addition to virtual environments, remapping has also given rise to new applications—immersive teleoperated robots. This paper selects immersive telesurgery, a representative scenario, as an opportunity for research, exploring the generalized effects of remapping. In such a scenario, the operator can observe through the robot's camera and use their hands to control the robotic arms, as if they were the robot. However, common remapping of spatial head-hand relations—due to camera adjustments and robotic arm switching—creates significant visual-proprioceptive conflicts and physical limitations. To explore this, we simulated a telesurgery system with 6 head-camera and 12 hand-robotic-arm remapping conditions, assessing non-surgeon participants across four surgical tasks: navigation, location, cutting, and bimanual coordination. The study examines spatial perception bias, interaction deviation, workload, and task completion time. Our findings reveal how different remapping targets, attributes, intensities, and situations affect performance, contributing to the understanding of perception mechanisms and offering insights for optimizing operations or systems.2025TLTianren Luo et al.Institute of Software, Chinese Academy of Sciences; College of Computer Science and Technology, University of Chinese Academy of SciencesTeleoperated DrivingHuman-Robot Collaboration (HRC)CHI
"Did you sleep well?": A Multimodal Sleep Diary for Sustained Self-Reporting by ChildrenSleep diaries are essential self-reporting tools for understanding children's sleep patterns, but maintaining sustained engagement and high-quality self-reporting remains challenging. While voice input has been explored in child-computer interaction research as a method to improve engagement, limited evidence exists regarding its effectiveness in supporting sustained self-reporting over time. To address this gap, we conducted a five-day field study with 20 children aged seven to twelve, using a multimodal sleep diary that integrated both voice and text input modalities. Our findings reveal that voice input significantly supports younger children in maintaining engagement over five days, though their response quality remains lower than that of older children. Two distinct response quality patterns over time also emphasize the importance of accounting for individual differences in task performance. Furthermore, input modality preferences varied by age: older children consistently favored text input, while younger children generally preferred voice input over time. These results highlight the potential of incorporating voice input into text-based sleep diaries to better accommodate the diverse needs of children, enhancing both sustained engagement and response quality. Future studies with longer observation periods are needed to validate and extend these findings.2025SCShanshan Chen et al.Eindhoven University of Technology, Department of Industrial DesignSleep & Stress MonitoringKnowledge Worker Tools & WorkflowsParticipatory DesignCHI
VAction: A Lightweight and Integrated VR Training System for Authentic Film-Shooting ExperienceThe film industry exerts significant economic and cultural influence, and its rapid development is contingent upon the expertise of industry professionals, underscoring the critical importance of film-shooting education. However, this process typically necessitates multiple practice in complex professional venues using expensive equipment, presenting a significant obstacle for ordinary learners who struggle to access such training environments. Despite VR technology has already shown its potential in education, existing research has not addressed the crucial learning component of replicating the shooting process. Moreover, the limited functionality of traditional controllers hinder the fulfillment of the educational requirements. Therefore, we developed VAction VR system, combining high-fidelity virtual environments with a custom-designed controller to simulate the real-world camera operation experience. The system’s lightweight design ensures cost-effective and efficient deployment. Experiment results demonstrated that VAction significantly outperforms traditional methods in both practice effectiveness and user experience, indicating its potential and usefulness in film-shooting education.2025SWShaocong Wang et al.Tsinghua University, Department of Computer Science and TechnologyMixed Reality WorkspacesHome Energy ManagementCHI
Emotionally Challenging Games Can Satisfy Older Adults' Psychological Needs: From Empirical Study to Design GuidelinesOlder adults often struggle to meet their psychological needs due to retirement and living alone. Recent studies suggest that games featuring emotional challenge (EC) can help fulfill basic psychological needs such as autonomy, competence, and relatedness by facilitating emotional exploration. However, it remains unclear whether older adults can benefit from EC games, whether they find this genre enjoyable, and how these games should be designed to better meet their needs. This work explores older adults’ experiences and perceptions of playing EC games through two studies. The first study involved playing Detroit: Become Human, revealing that older adults derived multifaceted psychological experiences from playing the game. The second study involved a custom-designed game scenario tailored to older adults, demonstrating that meaningful choices significantly influenced autonomy need satisfaction. Based on these findings, we offer five design guidelines for developing EC games that satisfy psychological needs of older adults.2025MZMin Zhou et al.Institute of Software, ChineseAging-Friendly Technology DesignSerious & Functional GamesCHI
Slip-Grip: An Electrotactile Method to Simulate WeightWeight perception is crucial for immersive virtual reality (VR) interactions, yet providing weight feedback remains a significant research challenge. We introduce a novel weight simulation technique that leverages electrotactile stimulation to induce slip illusions. These slip illusions occur when users grip an object with less force than a predefined threshold, allowing the device to modulate the grip force and encourage a tighter grip. In our approach, heavier virtual weights correspond to higher required grip forces. We conducted a series of user experiments to validate our technique, confirming that it effectively induces slip illusions. We also investigated the relationship between electrotactile sensations and grip force, and changes in force, demonstrating that this association enhances the weight perception experience. Lastly, we explored the mapping between grip force and perceived weight, observing strong linearity within participants but notable variability between individuals.2025HLHongnan Lin et al.Institute of Software, Chinese Academy of SciencesForce Feedback & Pseudo-Haptic WeightElectrical Muscle Stimulation (EMS)CHI
RemapVR: An Immersive Authoring Tool for Rapid Prototyping of Remapped Interaction in VRRemapping techniques in VR such as repositioning, redirection, and resizing have been extensively studied. Still, interaction designers rarely have the opportunity to use them due to high technical and knowledge barriers. In the paper, we extract common features of 24 existing remapping techniques and develop a high-fidelity immersive authoring tool, namely RemapVR, for rapidly building and experiencing prototypes of remapped space properties in VR that are unperceivable or acceptable to users. RemapVR provides designers with a series of functions for editing remappings and visualizing spatial property changes, mapping relationships between real and virtual worlds, sensory conflicts, etc. Designers can quickly build existing remappings via templates, and author new remappings by interactively recording spatial relations between input trajectory in real world and output trajectory in virtual world. User studies showed that the designs of RemapVR can effectively improve designers' authoring experience and efficiency, and support designers to author remapping prototypes that meet scene requirements and provide good user experience.2025TLTianren Luo et al.Institute of Software, Chinese Academy of Sciences; College of Computer Science and Technology, University of Chinese Academy of SciencesMixed Reality WorkspacesPrototyping & User TestingCHI
TutorCraftEase: Enhancing Pedagogical Question Creation with Large Language ModelsPedagogical questions are crucial for fostering student engagement and learning. In daily teaching, teachers pose hundreds of questions to assess understanding, enhance learning outcomes, and facilitate the transfer of theory-rich content. However, even experienced teachers often struggle to generate a large volume of effective pedagogical questions. To address this, we introduce TutorCraftEase, an interactive generation system that leverages large language models (LLMs) to assist teachers in creating pedagogical questions. TutorCraftEase enables the rapid generation of questions at varying difficulty levels with a single click, while also allowing for manual review and refinement. In a comparative user study with 39 participants, we evaluated TutorCraftEase against a traditional manual authoring tool and a basic LLM tool. The results show that TutorCraftEase can generate pedagogical questions comparable in quality to those created by experienced teachers, while significantly reducing their workload and time.2025WKWenhui Kang et al.University of Chinese Academy of Sciences; Institute of Software, Chinese Academy of Sciences, Beijing Key Laboratory of Human-Computer InteractionHuman-LLM CollaborationOnline Learning & MOOC PlatformsIntelligent Tutoring Systems & Learning AnalyticsCHI
Beyond Explicit and Implicit: How Users Provide Feedback to Shape Personalized Recommendation ContentAs personalized recommendation algorithms become integral to social media platforms, users are increasingly aware of their ability to influence recommendation content. However, limited research has explored how users provide feedback through their behaviors and platform mechanisms to shape the recommendation content. We conducted semi-structured interviews with 34 active users of algorithmic-driven social media platforms (e.g., Xiaohongshu, Douyin). In addition to explicit and implicit feedback, this study introduced intentional implicit feedback, highlighting the actions users intentionally took to refine recommendation content through perceived feedback mechanisms. Additionally, choices of feedback behaviors were found to align with specific purposes. Explicit feedback was primarily used for feed customization, while unintentional implicit feedback was more linked to content consumption. Intentional implicit feedback was employed for multiple purposes, particularly in increasing content diversity and improving recommendation relevance. This work underscores the user intention dimension in the explicit-implicit feedback dichotomy and offers insights for designing personalized recommendation feedback that better responds to users' needs.2025WLWenqi Li et al.Peking University, Department of Information ManagementExplainable AI (XAI)Recommender System UXCHI
Application of Prompt Learning Models in Identifying the Collaborative Problem Solving Skills in an Online TaskCollaborative problem solving (CPS) competence is considered one of the essential 21st-century skills. To facilitate the assessment and learning of CPS competence, researchers have proposed a series of frameworks to conceptualize CPS and explored ways to make sense of the complex processes involved in collaborative problem solving. However, encoding explicit behaviors into subskills within the frameworks of CPS skills is still a challenging task. Traditional studies have relied on manual coding to decipher behavioral data for CPS, but such coding methods can be very time-consuming and cannot support real-time analyses. Scholars have begun to explore approaches for constructing automatic coding models. Nevertheless, the existing models built using machine learning or deep learning techniques depend on a large amount of training data and have relatively low accuracy. To address these problems, this paper proposes a prompt-based learning pre-trained model. The model can achieve high performance even with limited training data. In this study, three experiments were conducted, and the results showed that our model not only produced the highest accuracy, macro F1 score, and kappa values on large training sets, but also performed the best on small training sets of the CPS behavioral data. The application of the proposed prompt-based learning pre-trained model contributes to the CPS skills coding task and can also be used for other CSCW coding tasks to replace manual coding.2024MZMengxiao Zhu et al.Session 2a: Collaborative WorkflowsCSCW
Model Touch Pointing and Detect Parkinson's Disease via a Mobile GameLing 等人开发基于移动游戏的触控点建模方法，通过分析玩家在游戏中的触控行为特征，实现帕金森病的早期辅助检测，为疾病筛查提供新途径。2024KLKaiyan Ling et al.Motor Impairment Assistive Input TechnologiesSerious & Functional GamesUbiComp
SpeciFingers: Finger Identification and Error Correction on Capacitive TouchscreensHuang 等人提出 SpeciFingers 系统，通过分析电容触摸屏信号特征实现多手指识别与错误纠正，提升触控交互准确性。2024ZHZeyuan Huang et al.Hand Gesture RecognitionContext-Aware ComputingUbiComp
Hypnos: A Contactless Sleep Stage Monitoring System Using UWB SignalsLi 等人提出 Hypnos 系统，采用无接触超宽带信号技术，实时监测用户睡眠阶段。2024SLSiheng Li et al.Sleep & Stress MonitoringBiosensors & Physiological MonitoringUbiComp
Waffle: A Waterproof mmWave-based Human Sensing System inside Bathrooms with Running WaterZhang 等人开发 Waffle 防水毫米波传感系统，专门解决浴室有流水环境中的人体感知难题，实现全天候室内监测。2024XZXusheng Zhang et al.Human Pose & Activity RecognitionContext-Aware ComputingUbiComp
Push the Limit of Highly Accurate Ranging on Commercial UWB DevicesMa 等人提出针对商业 UWB 设备的高精度测距优化方案，突破现有技术极限，提升室内定位精度。2024JMJunqi Ma et al.Context-Aware ComputingUbiquitous ComputingUbiComp
Embracing Distributed Acoustic Sensing in Car Cabin for Children Presence DetectionSu 等人利用分布式声学传感技术检测车内儿童存在状态，通过声学特征识别防止儿童被遗忘在车内，提升乘车安全。2024YSYuqi Su et al.In-Vehicle Haptic, Audio & Multimodal FeedbackMotion Sickness & Passenger ExperienceV2X (Vehicle-to-Everything) Communication DesignUbiComp
Exploring the Effects of Sensory Conflicts on Cognitive Fatigue in VR RemappingsVirtual reality (VR) is found to present significant cognitive challenges due to its immersive nature and frequent sensory conflicts. This study systematically investigates the impact of sensory conflicts induced by VR remapping techniques on cognitive fatigue, and unveils their correlation. We utilized three remapping methods (haptic repositioning, head-turning redirection, and giant resizing) to create different types of sensory conflicts, and measured perceptual thresholds to induce various intensities of the conflicts. Through experiments involving cognitive tasks along with subjective and physiological measures, we found that all three remapping methods influenced the onset and severity of cognitive fatigue, with visual-vestibular conflict having the greatest impact. Interestingly, visual-experiential/memory conflict showed a mitigating effect on cognitive fatigue, emphasizing the role of novel sensory experiences. This study contributes to a deeper understanding of cognitive fatigue under sensory conflicts and provides insights for designing VR experiences that align better with human perceptual and cognitive capabilities.2024TLTianren Luo et al.Eye Tracking & Gaze InteractionImmersion & Presence ResearchUIST