"If I were in Space": Understanding and Adapting to Social Isolation through Designing Collaborative StorytellingSocial isolation can lead to pervasive health issues like anxiety and loneliness. Previous work focused on physical interventions like exercise and teleconferencing, but overlooked the narrative potential of adaptive strategies. To address this, we designed a collaborative online storytelling experience in social VR, enabling participants in isolation to design an imaginary space journey as a metaphor for quarantine, in order to learn about their isolation adaptation strategies in the process. Eighteen individuals participated during real quarantine undertaken a virtual role-play experience, designing their own spaceship rooms and engaging in collaborative activities that revealed creative adaptative strategies. Qualitative analyses of participant designs, transcripts, and interactions revealed how they coped with isolation, and how the engagement unexpectedly influenced their adaptation process. This study shows how designing playful narrative experiences, rather than solution-driven approaches, can serve as probes to surface how people navigate social isolation.2025QGQi Gong et al.Social & Collaborative VRIdentity & Avatars in XRSTEM Education & Science CommunicationDIS
EyeSee: Enhancing Art Appreciation through Anthropomorphic Interpretations from Multiple PerspectivesArt appreciation serves as a crucial medium for emotional communication and sociocultural dialogue. In the digital era, fostering deep user engagement on online art appreciation platforms remains a challenge. Leveraging large language models (LLMs), we present EyeSee, a system designed to engage users through anthropomorphic characters. We implemented and evaluated three modes--Narrator, Artist, and In-Situ--acting as a third-person narrator, a first-person creator, and first-person created objects, respectively, across two sessions: Narrative and Recommendation. We conducted a within-subject study with 24 participants. In the Narrative session, we found that the In-Situ and Artist modes had higher aesthetic appeal than the Narrator mode, although the Artist mode showed lower perceived usability. Additionally, from the Narrative to the Recommendation session, we found that the user-perceived relatability and believability were sustained, but the user-perceived consistency and stereotypicality changed. Our findings suggest novel implications for anthropomorphic character design in enhancing user engagement.2025YLYongming Li et al.Xi'an Jiaotong University, MOE KLINNS LabAgent Personality & AnthropomorphismGenerative AI (Text, Image, Music, Video)Human-LLM CollaborationCHI
Learning from User-driven Events to Generate Automation SequencesSong 等人提出从用户驱动事件中学习生成自动化序列的方法，通过分析用户操作模式创建个性化工作流，提升智能系统自动化水平。2024YSYunpeng Song et al.Smart Home Interaction DesignUser Research Methods (Interviews, Surveys, Observation)UbiComp
Understanding the Effects of Restraining Finger Coactivation in Mid-Air Typing: from a Neuromechanical PerspectiveTyping in mid-air is often perceived as intuitive yet presents challenges due to finger coactivation, a neuromechanical phenomenon that involves involuntary finger movements stemming from the lack of physical constraints. Previous studies were used to examine and address the impacts of finger coactivation using algorithmic approaches. Alternatively, this paper explores the neuromechanical effects of finger coactivation on mid-air typing, aiming to deepen our understanding and provide valuable insights to improve these interactions. We utilized a wearable device that restrains finger coactivation as a prop to conduct two mid-air studies, including a rapid finger-tapping task and a ten-finger typing task. The results revealed that restraining coactivation not only reduced mispresses, which is a classic coactivated error always considered as harm caused by coactivation. Unexpectedly, the reduction of motor control errors and spelling errors, thinking as non-coactivated errors, also be observed. Additionally, the study evaluated the neural resources involved in motor execution using functional Near Infrared Spectroscopy (fNIRS), which tracked cortical arousal during mid-air typing. The findings demonstrated decreased activation in the primary motor cortex of the left hemisphere when coactivation was restrained, suggesting a diminished motor execution load. This reduction suggests that a portion of neural resources is conserved, which also potentially aligns with perceived lower mental workload and decreased frustration levels.2024HZHechuan Zhang et al.Full-Body Interaction & Embodied InputUIST
NotePlayer: Engaging Jupyter Notebooks for Dynamic Presentation of Analytical ProcessesDiverse presentation formats play a pivotal role in effectively conveying code and analytical processes during data analysis. One increasingly popular format is tutorial videos, particularly those based on Jupyter notebooks, which offer an intuitive interpretation of code and vivid explanations of analytical procedures. However, creating such videos requires a diverse skill set and significant manual effort, posing a barrier for many analysts. To bridge this gap, we introduce an innovative tool called NotePlayer, which connects notebook cells to video segments and incorporates a computational engine with language models to streamline video creation and editing. Our aim is to make the process more accessible and efficient for analysts. To inform the design of NotePlayer, we conducted a formative study and performed content analysis on a corpus of 38 Jupyter tutorial videos. This helped us identify key patterns and challenges encountered in existing tutorial videos, guiding the development of NotePlayer. Through a combination of a usage scenario and a user study, we validated the effectiveness of NotePlayer. The results show that the tool streamlines the video creation and facilitates the communication process for data analysts.2024YOYang Ouyang et al.Prototyping & User TestingComputational Methods in HCIUIST
VisionTasker: Mobile Task Automation Using Vision Based UI Understanding and LLM Task PlanningMobile task automation is an emerging field that leverages AI to streamline and optimize the execution of routine tasks on mobile devices, thereby enhancing efficiency and productivity. Traditional methods, such as Programming By Demonstration (PBD), are limited due to their dependence on predefined tasks and susceptibility to app updates. Recent advancements have utilized the view hierarchy to collect UI information and employed Large Language Models (LLM) to enhance task automation. However, view hierarchies have accessibility issues and face potential problems like missing object descriptions or misaligned structures. This paper introduces VisionTasker, a two-stage framework combining vision-based UI understanding and LLM task planning, for mobile task automation in a step-by-step manner. VisionTasker firstly converts a UI screenshot into natural language interpretations using a vision-based UI understanding approach, eliminating the need for view hierarchies. Secondly, it adopts a step-by-step task planning method, presenting one interface at a time to the LLM. The LLM then identifies relevant elements within the interface and determines the next action, enhancing accuracy and practicality. Extensive experiments show that VisionTasker outperforms previous methods, providing effective UI representations across four datasets. Additionally, in automating 147 real-world tasks on an Android smartphone, VisionTasker demonstrates advantages over humans in tasks where humans show unfamiliarity and shows significant improvements when integrated with the PBD mechanism. VisionTasker is open-source and available at https://github.com/AkimotoAyako/VisionTasker.2024YSYunpeng Song et al.Human-LLM CollaborationAI-Assisted Decision-Making & AutomationContext-Aware ComputingUIST
Towards Building Condition-Based Cross-Modality Intention-Aware Human-AI Cooperation under VR EnvironmentTo address critical challenges in effectively identifying user intent and forming relevant information presentations and recommendations in VR environments, we propose an innovative condition-based multi-modal human-AI cooperation framework. It highlights the intent tuples (intent, condition, intent prompt, action prompt) and 2-Large-Language-Models (2-LLMs) architecture. This design, utilizes ``condition'' as the core to describe tasks, dynamically match user interactions with intentions, and empower generations of various tailored multi-modal AI responses. The architecture of 2-LLMs separates the roles of intent detection and action generation, decreasing the prompt length and helping with generating appropriate responses. We implemented a VR-based intelligent furniture purchasing system based on the proposed framework and conducted a three-phase comparative user study. The results conclusively demonstrate the system's superiority in time efficiency and accuracy, intention conveyance improvements, effective product acquisitions, and user satisfaction and cooperation preference. Our framework provides a promising approach towards personalized and efficient user experiences in VR.2024ZHZiyao He et al.Xi'an Jiaotong UniversitySocial & Collaborative VRAR Navigation & Context AwarenessHuman-LLM CollaborationCHI
TacTex: A Textile Interface with Seamlessly-Integrated Electrodes for High-Resolution electrotactile StimulationThis paper presents TacTex, a textile-based interface that provides high-resolution haptic feedback and touch-tracking capabilities. TacTex utilizes electrotactile stimulation, which has traditionally posed challenges due to limitations in textile electrode density and quantity. TacTex overcomes these challenges by employing a multi-layer woven structure that separates conductive weft and warp electrodes with non-conductive yarns. The driving system for TacTex includes a power supply, sensing board, and switch boards to enable spatial and temporal control of electrical stimuli on the textile, while simultaneously monitoring voltage changes. TacTex can stimulate a wide range of haptic effects, including static and dynamic patterns and different sensation qualities, with a resolution of $512 \times 512$ and \textcolor{black}{based on linear electrodes spaced as closely as 2mm}. We evaluate the performance of the interface with user studies and demonstrate the potential applications of TacTex interfaces in everyday textiles for adding haptic feedback.2024HLHongnan Lin et al.Institute of Software, Chinese Academy of SciencesVibrotactile Feedback & Skin StimulationShape-Changing Interfaces & Soft Robotic MaterialsElectronic Textiles (E-textiles)CHI
Data Cubes in Hand: A Design Space of Tangible Cubes for Visualizing 3D Spatio-Temporal Data in Mixed RealityTangible interfaces in mixed reality (MR) environments allow for intuitive data interactions. Tangible cubes, with their rich interaction affordances, high maneuverability, and stable structure, are particularly well-suited for exploring multi-dimensional data types. However, the design potential of these cubes is underexplored. This study introduces a design space for tangible cubes in MR, focusing on interaction space, visualization space, sizes, and multiplicity. Using spatio-temporal data, we explored the interaction affordances of these cubes in a workshop (N=24). We identified unique interactions like rotating, tapping, and stacking, which are linked to augmented reality (AR) visualization commands. Integrating user-identified interactions, we created a design space for tangible-cube interactions and visualization. A prototype visualizing global health spending with small cubes was developed and evaluated, supporting both individual and combined cube manipulation. This research enhances our grasp of tangible interaction in MR, offering insights for future design and application in diverse data contexts.2024SHShuqi He et al.Xi'an Jiaotong - Liverpool UniversityMixed Reality WorkspacesInteractive Data VisualizationTime-Series & Network Graph VisualizationCHI
Integrating Gaze and Mouse Via Joint Cross-Attention Fusion Net for Students’ Activity Recognition in E-learning"E-learning has emerged as an indispensable educational mode in the post-epidemic era. However, this mode makes it difficult for students to stay engaged in learning without appropriate activity monitoring. Our work explores a promising solution that combines gaze and mouse data to recognize students' activities, thereby facilitating activity monitoring and analysis during e-learning. We initially surveyed 200 students from a local university, finding more acceptance for eye trackers and mouse loggers compared to video surveillance. We then designed eight students' routine digital activities to collect a multimodal dataset and analyze the patterns and correlations between gaze and mouse across various activities. Our proposed Joint Cross-Attention Fusion Net, a multimodal activity recognition framework, leverages the gaze-mouse relationship to yield improved classification performance by integrating cross-modal representations through a cross-attention mechanism and integrating the joint features that characterize gaze-mouse coordination. Evaluation results show that our method can achieve up to 94.87% F1-score in predicting 8-classes activities, with an improvement of at least 7.44% over using gaze or mouse data independently. This research illuminates new possibilities for monitoring student engagement in intelligent education systems, also suggesting a promising strategy for melding perception and action modalities in behavioral analysis across a range of ubiquitous computing environments." https://doi.org/10.1145/36108762023RZRongrong Zhu et al.Eye Tracking & Gaze InteractionIntelligent Tutoring Systems & Learning AnalyticsUbiComp
Interaction of Thoughts: Towards Mediating Task Assignment in Human-AI Cooperation with a Capability-Aware Shared Mental ModelThe existing work on task assignment of human-AI cooperation did not consider the differences between individual team members regarding their capabilities, leading to sub-optimal task completion results. In this work, we propose a capability-aware shared mental model (CASMM) with the components of task grouping and negotiation, which utilize tuples to break down tasks into sets of scenarios relating to difficulties and then dynamically merge the task grouping ideas raised by human and AI through negotiation. We implement a prototype system and a 3-phase user study for the proof of concept via an image labeling task. The result shows building CASMM boosts the accuracy and time efficiency significantly through forming the task assignment close to real capabilities within few iterations. It helps users better understand the capability of AI and themselves. Our method has the potential to generalize to other scenarios such as medical diagnoses and automatic driving in facilitating better human-AI cooperation.2023ZHZiyao He et al.Xi'an Jiaotong UniversityHuman-LLM CollaborationAI-Assisted Decision-Making & AutomationComputational Methods in HCICHI
Datavoidant: An AI System for Addressing Political Data Voids on Social MediaThe limited information (data voids) on political topics relevant to underrepresented communities has facilitated the spread of disinformation. Independent journalists who combat disinformation in underrepresented communities have reported feeling overwhelmed because they lack the tools necessary to make sense of the information they monitor and address the data voids. In this paper, we present a system to identify and address political data voids within underrepresented communities. Armed with an interview study, indicating that the independent news media has the potential to address them, we designed an intelligent collaborative system, called Datavoidant. Datavoidant uses state-of-the-art machine learning models and introduces a novel design space to provide independent journalists with a collective understanding of data voids to facilitate generating content to cover the voids. We performed a user interface evaluation with independent news media journalists (N=22). These journalists reported that Datavoidant's features allowed them to more rapidly while easily having a sense of what was taking place in the information ecosystem to address the data voids. They also reported feeling more confident about the content they created and the unique perspectives they had proposed to cover the voids. We conclude by discussing how Datavoidant enables a new design space wherein individuals can collaborate to make sense of their information ecosystem and actively devise strategies to prevent disinformation.2022CFClaudia Flores-Saviaga et al.Misinformation, Fake News & Conspiracy TheoriesCSCW
Datavoidant: An AI System for Addressing Political Data Voids on Social MediaThe limited information (data voids) on political topics relevant to underrepresented communities has facilitated the spread of disinformation. Independent journalists who combat disinformation in underrepresented communities have reported feeling overwhelmed because they lack the tools necessary to make sense of the information they monitor and address the data voids. In this paper, we present a system to identify and address political data voids within underrepresented communities. Armed with an interview study, indicating that the independent news media has the potential to address them, we designed an intelligent collaborative system, called Datavoidant. Datavoidant uses state-of-the-art machine learning models and introduces a novel design space to provide independent journalists with a collective understanding of data voids to facilitate generating content to cover the voids. We performed a user interface evaluation with independent news media journalists (N=22). These journalists reported that Datavoidant's features allowed them to more rapidly while easily having a sense of what was taking place in the information ecosystem to address the data voids. They also reported feeling more confident about the content they created and the unique perspectives they had proposed to cover the voids. We conclude by discussing how Datavoidant enables a new design space wherein individuals can collaborate to make sense of their information ecosystem and actively devise strategies to prevent disinformation.2022CFClaudia Flores-Saviaga et al.Misinformation, Fake News & Conspiracy TheoriesCSCW
HapTag: A Compact Actuator for Rendering Push-Button Tactility on Soft SurfacesAs touch interactions become ubiquitous in the field of human computer interactions, it is critical to enrich haptic feedback to improve efficiency, accuracy and immersive experiences. This paper presents HapTag, a thin and flexible actuator to support integration of push button tactile renderings to daily soft surfaces. Specifically, HapTag works under the principle of hydraulically amplified electroactive actuator (HASEL) while being optimized by embedding a pressure sensing layer, and being activated with dedicated voltage appliance in response to users' input actions, resulting in fast response time, controllable and expressive push-button tactile rendering capabilities. HapTag is in compact formfactor, and can be attached, integrated, or embedded on various soft surfaces like cloth, leather and rubber. Three common push button tactile patterns were adopted and implemented with HapTag. We validated the feasibility and expressiveness of HapTag by demonstrating a series of innovative applications under different circumstances.2022YCYanjun Chen et al.Vibrotactile Feedback & Skin StimulationHaptic WearablesShape-Changing Interfaces & Soft Robotic MaterialsUIST
vMirror: Enhancing the Interaction with Occluded or Distant Objects in VR with Virtual MirrorsInteracting with out of reach or occluded VR objects can be cumbersome. Although users can change their position and orientation, such as via teleporting, to help observe and select, doing so frequently may cause loss of spatial orientation or motion sickness. We present vMirror, an interactive widget leveraging reflection of mirrors to observe and select distant or occluded objects. We first designed interaction techniques for placing mirrors and interacting with objects through mirrors. We then conducted a formative study to explore a semi-automated mirror placement method with manual adjustments. Next, we conducted a target-selection experiment to measure the effect of the mirror's orientation on users' performance. Results showed that vMirror can be as efficient as direct target selection for most mirror orientations. We further compared vMirror with teleport technique in a virtual treasure hunt game and measured participants’ task performance and subjective experiences. Finally, we discuss vMirorr user experience and present future directions.2021NLNianlong Li et al.Institute of Software, Chinese Academy of Sciences, Institute of Software, Chinese Academy of SciencesSocial & Collaborative VRImmersion & Presence ResearchCHI
Does Trait Loneliness Predict Rejection of Social Robots? The Role of Reduced Attributions of Unique Humanness Since chronic loneliness is both a painful individual experience and an increasingly serious social problem, robot companions have emerged as a result of robotization of social work to confront this issue. We foresee that social robots will become pervasive in the near future. Thus, it is crucial to pinpoint the relationship between chronic experiences of loneliness (i.e., trait loneliness) and both anthropomorphism and acceptance of such artificial intelligent agents. Previous research demonstrated that experimentally induced state loneliness increases anthropomorphic inferences about nonhuman agents such as pets. However, in the present research we found that trait (vs. state) loneliness — a permanent personality disposition that is not easily relieved (vs. transitory experiences caused by circumstance, and easily relieved) — reduced participants’ anthropomorphic tendencies and acceptance of a social robot (regardless of the form: a picture of the robot, an on-site robot, or direct interaction with the robot). In particular, believing that the robot lacks good “unique humanness” traits (i.e., Humble, Thorough, Organized, Broadminded, and Polite) is one reason why dispositionally lonely participants are less likely to anthropomorphize a robot, which further prompts reduced acceptance of it. This finding suggests that unique humanness, exemplifying secondary emotions, is vital, not only in interpersonal contexts, but in establishing connections with social robots.2020SLSijia Li et al.Agent Personality & AnthropomorphismSocial Robot InteractionHRI
I'm All Eyes and Ears: Exploring Effective Locators for Privacy Awareness in IoT ScenariosWith the proliferation of IoT devices, there are growing concerns about being sensed or monitored by these devices unawares, especially in places perceived as private. We explore the design space of IoT locators to help people physically find nearby IoT devices. We first conducted a survey to understand people's willingness, current practices, and challenges in finding IoT devices. Our survey findings motivated us to design and implement low-cost locators (visual, auditory, and contextualized pictures) to help people find nearby devices. Through an iterative design process and two rounds of experiments, we found that these locators greatly reduced people's search time over a baseline of no locators. Many participants found the visual and auditory locators enjoyable. Some participants also appropriated the use of our system for other purposes, e.g., to learn about new IoT devices, instead of for privacy awareness.2020YSYunpeng Song et al.Xi'an Jiaotong UniversityPrivacy by Design & User ControlIoT Device PrivacyCHI
Normal and Easy: Account Sharing Practices in the WorkplaceWork is being digitized across all sectors, and digital account sharing has become common in the workplace. In this paper, we conduct a qualitative and quantitative study of digital account sharing practices in the workplace. Across two surveys, we examine the sharing process at work, probing what accounts people share, how and why they share those accounts, and identifying the major challenges people face in sharing accounts. Our results demonstrate that account sharing in the modern workplace serves as a norm rather than a simple workaround; centralizing collaborative activity and reducing boundary management effort are key motivations for sharing. But people still struggle with a lack of activity accountability and awareness, conflicts over simultaneous access, difficulties controlling access, and collaborative password use. Our work provides insights into the current difficulties people face in workplace collaboration with online account sharing, as a result of inappropriate designs that still assume a single-user model for accounts. We highlight opportunities for CSCW and HCI researchers and designers to better support sharing by multiple people in a more usable and secure way.2019YSYunpeng Song et al.WorkplacesCSCW