EchoAid: Enhancing Livestream Shopping Accessibility for the DHH CommunityLivestream shopping platforms often overlook the accessibility needs of the Deaf and Hard of Hearing (DHH) community, leading to barriers such as information inaccessibility and overload. To tackle these challenges, we developed EchoAid, a mobile app designed to improve the livestream shopping experience for DHH users. EchoAid utilizes advanced speech-to-text conversion, Rapid Serial Visual Presentation (RSVP) technology, and Large Language Models (LLMs) to simplify the complex information flow in live sales environments. We conducted exploratory studies with eight DHH individuals to identify design needs and iteratively developed the EchoAid prototype based on feedback from three participants. We then evaluate the performance of this system in a user study workshop involving 38 DHH participants. Our findings demonstrate the successful design and validation process of EchoAid, highlighting its potential to enhance product information extraction, leading to reduced cognitive overload and more engaging and customized shopping experiences for DHH users.2025ZYZeyu Yang et al.Deaf and Hard-of-Hearing ResearchCSCW
AppAgent: Multimodal Agents as Smartphone UsersRecent advancements in large language models (LLMs) have led to the creation of intelligent agents capable of performing complex tasks. This paper introduces a novel LLM-based multimodal agent framework designed to operate smartphone applications. Our framework allows the agent to mimic human-like interactions such as tapping and swiping through a simplified action space, eliminating the need for system back-end access and enhancing its versatility across various apps. Central to the agent's functionality is an innovative in-context learning method, where it either autonomously explores or learns from human demonstrations, creating a knowledge base used to execute complex tasks across diverse applications. We conducted extensive testing with our agent on over 50 tasks spanning 10 applications, ranging from social media to sophisticated image editing tools. Additionally, a user study confirmed the agent's superior performance and practicality in handling a diverse array of high-level tasks, demonstrating its effectiveness in real-world settings. Our project page is available at \url{https://appagent-official.github.io/}.2025CZChi Zhang et al.Westlake University, School of EngineeringHuman-LLM CollaborationCHI
Investigating Context-Aware Collaborative Text Entry on Smartphones using Large Language ModelsText entry is a fundamental and ubiquitous task, but users often face challenges such as situational impairments or difficulties in sentence formulation. Motivated by this, we explore the potential of large language models (LLMs) to assist with text entry in real-world contexts. We propose a collaborative smartphone-based text entry system, CATIA, that leverages LLMs to provide text suggestions based on contextual factors, including screen content, time, location, activity, and more. In a 7-day in-the-wild study with 36 participants, the system offered appropriate text suggestions in over 80% of cases. Users exhibited different collaborative behaviors depending on whether they were composing text for interpersonal communication or information services. Additionally, the relevance of contextual factors beyond screen content varied across scenarios. We identified two distinct mental models: AI as a supportive facilitator or as a more equal collaborator. These findings outline the design space for human-AI collaborative text entry on smartphones.2025WCWeihao Chen et al.Tsinghua University, Department of Computer Science and TechnologyVoice User Interface (VUI) DesignHuman-LLM CollaborationContext-Aware ComputingCHI
CUPID: Improving Battle Fairness and Position Satisfaction in Online MOBA Games with a Re-matchmaking SystemThe multiplayer online battle arena (MOBA) genre has gained significant popularity and economic success, attracting considerable research interest within the Human-Computer Interaction community. Enhancing the gaming experience requires a deep understanding of player behavior, and a crucial aspect of MOBA games is matchmaking, which aims to assemble teams of comparable skill levels. However, existing matchmaking systems often neglect important factors such as players' position preferences and team assignment, resulting in imbalanced matches and reduced player satisfaction. To address these limitations, this paper proposes a novel framework called CUPID, which introduces a novel process called ``re-matchmaking'' to optimize team and position assignments to improve both fairness and player satisfaction. CUPID incorporates a pre-filtering step to ensure a minimum level of matchmaking quality, followed by a pre-match win-rate prediction model that evaluates the fairness of potential assignments. By simultaneously considering players' position satisfaction and game fairness, CUPID aims to provide an enhanced matchmaking experience. Extensive experiments were conducted on two large-scale, real-world MOBA datasets to validate the effectiveness of CUPID. The results surpass all existing state-of-the-art baselines, with an average relative improvement of 7.18% in terms of win prediction accuracy. Furthermore, CUPID has been successfully deployed in a popular online mobile MOBA game. The deployment resulted in significant improvements in match fairness and player satisfaction, as evidenced by critical Human-Computer Interaction (HCI) metrics covering usability, accessibility, and engagement, observed through A/B testing.2024GFGe Fan et al.Session 4f: Multiplayer Gaming and CommunicationCSCW
Push the Limit of Highly Accurate Ranging on Commercial UWB DevicesMa 等人提出针对商业 UWB 设备的高精度测距优化方案，突破现有技术极限，提升室内定位精度。2024JMJunqi Ma et al.Context-Aware ComputingUbiquitous ComputingUbiComp
Embracing Consumer-level UWB-equipped Devices for Fine-grained Wireless SensingRF sensing has been actively exploited in the past few years to enable novel IoT applications. Among different wireless technologies, WiFi-based sensing is most popular owing to the pervasiveness of WiFi infrastructure. However, one critical issue associated with WiFi sensing is that the information required for sensing can not be obtained from consumer-level devices such as smartphones or smart watches. The commonly-seen WiFi devices in our everyday lives actually can not be utilized for sensing. Instead, dedicated hardware with a specific WiFi card (e.g., Intel 5300) needs to be used for WiFi sensing. This paper involves Ultra-Wideband (UWB) into the ecosystem of RF sensing and makes RF sensing work on consumer-level hardware such as smartphones and smart watches for the first time. We propose a series of methods to realize UWB sensing on consumer-level electronics without any hardware modification. By leveraging fine-grained human respiration monitoring as the application example, we demonstrate that the achieved performance on consumer-level electronics is comparable to that achieved using dedicated UWB hardware. We show that UWB sensing hosted on consumer-level electronics is able to achieve fine granularity, robustness against interference and also multi-target sensing, pushing RF sensing one step towards real-life adoption. https://dl.acm.org/doi/10.1145/35694872023FZFusang Zhang et al.Biosensors & Physiological MonitoringContext-Aware ComputingUbiquitous ComputingUbiComp
MagKnitic: Machine-knitted Passive and Interactive Haptics Textiles with Integrated Binary SensingIn this paper, we introduce \textit{MagKnitic}, a novel approach to integrate passive force feedback and binary sensing into fabrics via digital machine knitting. Our approach utilizes digital fabrication technology to enable haptic interfaces that are soft, flexible, lightweight, and conform to the user's body shape. Despite these characteristics, our interfaces provide diverse, interactive, and responsive force feedback, expanding the design space for haptic experiences. \textit{MagKnitic} provides scalable and customizable passive haptic sensations by utilizing the attractive force between ferromagnetic yarns and permanent magnets, both of which are seamlessly integrated into knitted fabrics. Moreover, we present a binary sensing capability based on the resistance drop resulting from the activated electrical path between the integrated magnets and ferromagnetic yarn upon direct contact. We offer parametric design templates for users to customize \textit{MagKnitic} layouts and patterns. With various design layouts and combinations, \textit{MagKnitic} supports passive haptics interactions of linear, polar, angular, planar, radial, and user-defined motions. We perform a technical evaluation of the passive force feedback and the binary sensing capabilities with different machine knitting layouts and patterns, embedded magnet sizes, and interaction distances. In addition, we conduct two user studies to validate the effectiveness of \textit{MagKnitic}. Finally, we demonstrate various application scenarios, including wearable input interfaces, game controllers, passive VR/AR wearables, and interactive furniture coverings.2023YLYiyue Luo et al.Haptic WearablesShape-Changing Interfaces & Soft Robotic MaterialsUIST
MaraVis: Representation and Coordinated Intervention of Medical Encounters in Urban MarathonThere is an increased use of Internet-of-Things and wearable sensing devices in the urban marathon to ensure effective response to unforeseen medical needs. However, the massive amount of real-time, heterogeneous movement and psychological data of runners impose great challenges on prompt medical incident analysis and intervention. Conventional approaches compile such data into one dashboard visualization to facilitate rapid data absorption but fail to support joint decision-making and operations in medical encounters. In this paper, we present MaraVis, a real-time urban marathon visualization and coordinated intervention system. It first visually summarizes real-time marathon data to facilitate the detection and exploration of possible anomalous events. Then, it calculates an optimal camera route with an arrangement of shots to guide offline effort to catch these events in time with a smooth view transition. We conduct a within-subjects study with two baseline systems to assess the efficacy of MaraVis.2020QLQuan Li et al.WeBankTelemedicine & Remote Patient MonitoringSmart Cities & Urban SensingPublic Transit & Trip PlanningCHI
Understanding Motivations behind Inaccurate Check-insCheck-in data from social networks provide researchers a unique opportunity to model human dynamics at scale. However, it is unclear how indicative these check-in traces are of real human mobility. Prior work showed that a significant portion of Foursquare check-ins did not match with the physical mobility patterns of users, and suggested that misrepresented check-ins were incentivized by external rewards provided by the system. In this paper, our goal is to understand the root cause of inaccurate check-in data, by studying its validity in social media platforms without external rewards for check-ins. We conduct a data-driven analysis using an empirical check-in trace of more than 276,000 users from WeChat Moments, with matching traces of their physical mobility. We develop a set of hypotheses on the underlying motivations behind people's inaccurate check-ins, and validate them using a detailed user study which includes both surveys and interviews. Our analysis reveals that there are a surprisingly large number of inaccurate check-ins even in the absence of rewards: 43% of total check-ins are inaccurate and 61% of survey participants report they have misrepresented their check-ins. We also find that inaccurate check-ins are often a result of user interface design as well as for convenience, commercial advertisement and self-presentation.2018FXFengli Xu et al.Urban SpacesCSCW