EchoLadder: Progressive AI-Assisted Design of Immersive VR ScenesMixed reality platforms allow users to create virtual environments, yet novice users struggle with both ideation and execution in spatial design. While existing AI models can automatically generate scenes based on user prompts, the lack of interactive control limits users' ability to iteratively steer the output. In this paper, we present EchoLadder, a novel human-AI collaboration pipeline that leverages large vision-language model (LVLM) to support interactive scene modification in virtual reality. EchoLadder accepts users' verbal instructions at varied levels of abstraction and spatial specificity, generates concrete design suggestions throughout a progressive design process. The suggestions can be automatically applied, regenerated and retracted by users' toggle control. Our ablation study showed effectiveness of our pipeline components. Our user study found that, compared to end-to-end generation without showing suggestions, EchoLadder better supports user creativity in spatial design. It also contributes insights on users' progressive design strategies under AI assistance, providing design implications for future systems.2025ZHZhuangze Hou et al.Mixed Reality WorkspacesHuman-LLM CollaborationCreative Collaboration & Feedback SystemsUIST
RemapVR: An Immersive Authoring Tool for Rapid Prototyping of Remapped Interaction in VRRemapping techniques in VR such as repositioning, redirection, and resizing have been extensively studied. Still, interaction designers rarely have the opportunity to use them due to high technical and knowledge barriers. In the paper, we extract common features of 24 existing remapping techniques and develop a high-fidelity immersive authoring tool, namely RemapVR, for rapidly building and experiencing prototypes of remapped space properties in VR that are unperceivable or acceptable to users. RemapVR provides designers with a series of functions for editing remappings and visualizing spatial property changes, mapping relationships between real and virtual worlds, sensory conflicts, etc. Designers can quickly build existing remappings via templates, and author new remappings by interactively recording spatial relations between input trajectory in real world and output trajectory in virtual world. User studies showed that the designs of RemapVR can effectively improve designers' authoring experience and efficiency, and support designers to author remapping prototypes that meet scene requirements and provide good user experience.2025TLTianren Luo et al.Institute of Software, Chinese Academy of Sciences; College of Computer Science and Technology, University of Chinese Academy of SciencesMixed Reality WorkspacesPrototyping & User TestingCHI
Chorus of the Past: Toward Designing a Multi-agent Conversational Reminiscence System with Digital Artifacts for Older AdultsReminiscence has been shown to provide benefits for older adults, but traditionally relies on personal photos as memory cues and interactions with real people who may not always be available. We present ReminiBuddy, a novel LLM-powered multi-agent conversational system, which allows older adults to engage with two distinct agents—one embodying an older identity and the other a younger identity—while using not only personal photos but also 3D models of generic nostalgic objects as memory cues. Our study, with older adult participants, found that the conversational approach both enjoyable and beneficial for reminiscence. While the younger agent was perceived as more emotionally engaging, the older one fostered greater resonance in content. Personal photos prompted autobiographical memories, whereas 3D generic nostalgic objects evoked shared memories of an era, contributing to a more multifaceted reminiscence experience. We further present design implications for better supporting older adults in reminiscing with LLM-powered conversational agents.2025JSJingwei Sun et al.Lenovo ResearchHuman-LLM CollaborationElderly Care & Dementia SupportCHI
TouchEditor: Interaction Design and Evaluation of a Flexible Touchpad for Text Editing of Head-Mounted Displays in Speech-unfriendly EnvironmentsZhan 等人设计 TouchEditor 柔性触控板，解决语音不友好环境下头显显示器的文本编辑交互问题。2024LZLishuang Zhan et al.Head-Up Display (HUD) & Advanced Driver Assistance Systems (ADAS)Hand Gesture RecognitionVoice User Interface (VUI) DesignUbiComp
See Widely, Think Wisely: Toward Designing a Generative Multi-agent System to Burst Filter BubblesThe proliferation of AI-powered search and recommendation systems has accelerated the formation of "filter bubbles" that reinforce people's biases and narrow their perspectives. Previous research has attempted to address this issue by increasing the diversity of information exposure, which is often hindered by a lack of user motivation to engage with. In this study, we took a human-centered approach to explore how Large Language Models (LLMs) could assist users in embracing more diverse perspectives. We developed a prototype featuring LLM-powered multi-agent characters that users could interact with while reading social media content. We conducted a participatory design study with 18 participants and found that multi-agent dialogues with gamification incentives could motivate users to engage with opposing viewpoints. Additionally, progressive interactions with assessment tasks could promote thoughtful consideration. Based on these findings, we provided design implications with future work outlooks for leveraging LLMs to help users burst their filter bubbles.2024YZyu zhang et al.Southeast University, Lenovo ResearchHuman-LLM CollaborationAI-Assisted Decision-Making & AutomationAI Ethics, Fairness & AccountabilityCHI
Side-by-Side vs Face-to-Face: Evaluating Colocated Collaboration via a Transparent Wall-sized DisplayTraditional wall-sized displays mostly only support side-by-side co-located collaboration, while transparent displays naturally support face-to-face interaction. Many previous works assume transparent displays support collaboration. Yet it is unknown how exactly its afforded face-to-face interaction can support loose or close collaboration, especially compared to the side-by-side configuration offered by traditional large displays. In this paper, we used an established experimental task that operationalizes different collaboration coupling and layout locality, to compare pairs of participants collaborating side-by-side versus face-to-face in each collaborative situation. We compared quantitative measures and collected interview and observation data to further illustrate and explain our observed user behavior patterns. The results showed that the unique face-to-face collaboration brought by transparent display can result in more efficient task performance, different territorial behavior, and both positive and negative collaborative factors. Our findings provided empirical understanding about the collaborative experience supported by wall-sized transparent displays and shed light on its future design.2023JGJiangtao Gong et al.Collaboration ICSCW
Work with AI and Work for AI: Autonomous Vehicle Safety Drivers’ Lived ExperiencesThe development of Autonomous Vehicle (AV) has created a novel job, the safety driver, recruited from experienced drivers to supervise and operate AV in numerous driving missions. Safety drivers usually work with non-perfect AV in high-risk real-world traffic environments for road testing tasks. However, this group of workers is under-explored in the HCI community. To fill this gap, we conducted semi-structured interviews with 26 safety drivers. Our results present how safety drivers cope with defective algorithms and shape and calibrate their perceptions while working with AV. We found that, as front-line workers, safety drivers are forced to take risks accumulated from the AV industry upstream and are also confronting restricted self-development in working for AV development. We contribute the first empirical evidence of the lived experience of safety drivers, the first passengers in the development of AV, and also the grassroots workers for AV, which can shed light on future human-AI interaction research.2023MCMengdi Chu et al.Tsinghua UniversityAutomated Driving Interface & Takeover DesignAI-Assisted Decision-Making & AutomationCHI
Remote Co-teaching in Rural Classroom: Current Practices, Impacts, and ChallengesThe shortage of high-quality teachers is one of the biggest educational problems faced by underdeveloped areas. With the development of information and communication technologies (ICTs), China has begun a remote co-teaching intervention program using ICTs for rural classes, forming a unique "co-teaching classroom''. We conducted semi-structured interviews with nine remote urban teachers and twelve local rural teachers. We identified the remote co-teaching classes' standard practices and co-teachers' collaborative work process. We also found that remote teachers' high-quality class directly impacted local teachers and students. Furthermore, interestingly, local teachers were also actively involved in making indirect impacts on their students by deeply coordinating with remote teachers and adapting the resources offered by the remote teachers. We conclude by summarizing and discussing the challenges faced by teachers, lessons learned from the current program, and related design implications to achieve a more adaptive and sustainable ICT4D program design.2022SGSiling Guo et al.Lenovo ResearchSpecial Education TechnologyDeveloping Countries & HCI for Development (HCI4D)CHI
HoloBoard: a Large-format Immersive Teaching Board based on Pseudo HoloGraphicsIn this paper, we present HoloBoard, an interactive large-format pseduo-holographic display system for lecture based classes. With its unique properties of immersive visual display and transparent screen, we designed and implemented a rich set of novel interaction techniques like immersive presentation, role-play, and lecturing behind the scene that are potentially valuable for lecturing in class. We conducted a controlled experimental study to compare a HoloBoard class with a normal class through measuring students’ learning outcomes and three dimensions of engagement (i.e., behavioral, emotional, and cognitive engagement). We used pre-/post- knowledge tests and multimodal learning analytics to measure students’ learning outcomes and learning experiences. Results indicated that the lecture-based class utilizing HoloBoard lead to slightly better learning outcomes and a significantly higher level of student engagement. Given the results, we discussed the impact of HoloBoard as an immersive media in the classroom setting and suggest several design implications for deploying HoloBoard in immersive teaching practices.2021JGJiangtao Gong et al.Mixed Reality WorkspacesImmersion & Presence ResearchUIST
Critiquing for Music Exploration in Conversational Recommender SystemsDialogue-based conversational recommender systems allow users to give language-based feedback on the recommended item, which has great potential for supporting users to explore the space of recommendations through conversation. In this work, we consider incorporating critiquing techniques into conversational systems to facilitate users' exploration of music recommendations. Thus, we have developed a music chatbot with three system variants, which are respectively featured with three different critiquing techniques, i.e., user-initiated critiquing (UC), progressive system-suggested critiquing (Progressive SC), and cascading system-suggested critiquing (Cascading SC). We conducted a between-subject study (N=107) to compare these three types of systems with regards to music exploration in terms of user perception and user interaction. Results show that both UC and SC are useful for music exploration, while users perceive higher diversity of recommendations with the system that offers Cascading SC and perceive more serendipitous with the system that offers Progressive SC. In addition, we find that the critiquing techniques significantly moderate the relationships between some interaction metrics (e.g., number of listened songs, number of dialogue turns) and users' perceived helpfulness and serendipity during music exploration.2021WCWanling Cai et al.Conversational ChatbotsRecommender System UXIUI
All in One Group: Current Practices, Lessons and Challenges of Chinese Home-School Communication in IM Group ChatWhen schools and families form a good partnership, children benefit. With the recent flourishing of communication apps, families and schools in China have shifted their primary communication channels to chat groups hosted on popular instant-messenger(IM) tools such as WeChat and QQ. With an interview study consisting of 18 parents and 9 teachers, followed by a survey study with 210 teachers, we found that IM group chat has become the most popular way that the majority of parents and teachers communicate, from among the many different channels available. While there are definite advantages to this kind of group chat, we also found a number of problematic issues, including a lack of privacy and repeated negative feedback shared by both parents and teachers. We discuss our results on how IM-based group chat could affect Chinese teachers' authoritative figures, affect Chinese teacher's work-life balance and potentially compromise Chinese students' privacy.2021JGJiangtao Gong et al.Lenovo ResearchSocial Platform Design & User BehaviorGig Economy PlatformsUser Research Methods (Interviews, Surveys, Observation)CHI
ThermalRing: Gesture and Tag Inputs Enabled by a Thermal Imaging Smart RingThe heterogeneous and ubiquitous input demands in smart spaces call for an input device that can enable rich and spontaneous interactions. We propose ThermalRing, a thermal imaging smart ring using low-resolution thermal camera for identity-anonymous, illumination-invariant, and power-efficient sensing of both dynamic and static gestures. We also design ThermalTag, thin and passive thermal imageable tags that reflect the heat from the human hand. ThermalTag can be easily made and applied onto everyday objects by users. We develop sensing techniques for three typical input demands: drawing gestures for device pairing, click and slide gestures for device control, and tag scan gestures for quick access. The study results show that ThermalRing can recognize nine drawing gestures with an overall accuracy of 90.9%, detect click gestures with an accuracy of 94.9%, and identify among six ThermalTags with an overall accuracy of 95.0%. Finally, we show the versatility and potential of ThermalRing through various applications.2020TZTengxiang Zhang et al.Chinese Academy of Sciences & UCASHand Gesture RecognitionOn-Skin Display & On-Skin InputContext-Aware ComputingCHI