Towards Immersive Mixed Reality Street Play: Understanding Co-located Bodily Play with See-through Head-Mounted Displays in Public SpacesWe are witnessing an upcoming paradigm shift as Mixed Reality (MR) See-through Head-Mounted Displays (HMDs) become ubiquitous, with use shifting from controlled, private settings to spontaneous, public ones. While location-based pervasive mobile games like Pokémon GO have achieved success, the embodied interaction of MRHMDs is moving us from phone-based screen-touching gameplay to MRHMD-enabled co-located bodily play. Major tech companies are continuously releasing visionary videos where urban streets transform into vast MR playgrounds—imagine Harry Potter-style wizard duels on city streets. However, few researchers have conducted real-world, in-the-wild studies of such Immersive Mixed Reality Street Play (IMRSP) in public spaces in anticipation of a near future with prevalent MRHMDs. Through empirical studies on a series of research-through-design game probes called Multiplayer Omnipresent Fighting Arena (MOFA), we gain an initial understanding of this under-explored area by identifying the social implications, challenges, and opportunities of this new paradigm.2025BHBotao Amber Hu et al.Perspectives on VRCSCW
From Sports Videos to Immersive Training: Augmenting Human Motion to Enrich Basketball Training ExperienceVideo plays a crucial role in sports training, enabling participants to analyze their movements and identify opponents' weaknesses. Despite the easy access to sports videos, the rich motion data within them remains underutilized due to the lack of clear performance indicators and discrepancies from real-game conditions. To address this, we employed advanced computer vision algorithms to reconstruct human motions in an immersive environment, where users can freely observe and interact with the movements. Basketball shooting was chosen as a representative scenario to validate this framework, given its fast pace and extensive physical contact. Collaborating with experts, we iteratively designed motion-related visualizations to improve the understanding of complex movements. A one-on-one matchup simulating real games was also provided, allowing users to compete directly with the reconstructed motions. Our user studies demonstrate that this method enhances participants' movement comprehension and engagement, while insights derived from interviews inform future immersive training designs.2025YWYihong Wu et al.Full-Body Interaction & Embodied InputHuman Pose & Activity RecognitionUIST
CPVis: Evidence-based Multimodal Learning Analytics for Evaluation in Collaborative ProgrammingAs programming education becomes more widespread, many college students from non-computer science backgrounds begin learning programming. Collaborative programming emerges as an effective method for instructors to support novice students in developing coding and teamwork abilities. However, due to limited class time and attention, instructors face challenges in monitoring and evaluating the progress and performance of groups or individuals. To address this issue, we collect multimodal data from real-world settings and develop CPVis, an interactive visual analytics system designed to assess student collaboration dynamically. Specifically, CPVis enables instructors to evaluate both group and individual performance efficiently. CPVis employs a novel flower-based visual encoding to represent performance and provides time-based views to capture the evolution of collaborative behaviors. A within-subject experiment (N=22), comparing CPVis with two baseline systems, reveals that users gain more insights, find the visualization more intuitive, and report increased confidence in their assessments of collaboration.2025GZGefei Zhang et al.Zhejiang University of TechnologyInteractive Data VisualizationCollaborative Learning & Peer TeachingCHI
RouteFlow: Trajectory-Aware Animated TransitionsAnimating objects’ movements is widely used to facilitate tracking changes and observing both the global trend and local hotspots where objects converge or diverge. Existing methods, however, often obscure critical local hotspots by only considering the start and end positions of objects' trajectories. To address this gap, we propose RouteFlow, a trajectory-aware animated transition method that effectively balances the global trend and local hotspots while minimizing occlusion. RouteFlow is inspired by a real-world bus route analogy: objects are regarded as passengers traveling together, with local hotspots representing bus stops where these passengers get on and off. Based on this analogy, animation paths are generated like bus routes, with the object layout generated similarly to seat allocation according to their destinations. Compared with state-of-the-art methods, RouteFlow better facilitates identifying the global trend and locating local hotspots while performing comparably in tracking objects' movements.2025DLDuan Li et al.Tsinghua UniversityInteractive Data VisualizationVisualization Perception & CognitionCHI
Tendon Vibration for Creating Movement Illusions in Virtual RealityTendon vibration can create movement illusions: vibrating the biceps tendon induces an illusion of extending the arm, while vibrating the triceps tendon induces an illusion of flexing the arm. However, it is unclear how to create and integrate such illusions shown in neuroscience to interaction techniques in virtual reality (VR). We first design a motor setup for tendon vibration. Study 1 validates that the setup induces movement illusions which on average create a 5.26 cm offset in active arm movements. Study 2 shows that tendon vibration improves the detection thresholds of visual motion gains often used in VR interaction techniques by 0.22. A model we developed in Study 2 predicts the effects of tendon vibration and is used in a biomechanical simulation to demonstrate the detection thresholds across typical reaching tasks in VR.2025MCMantas Cibulskis et al.University of CopenhagenVibrotactile Feedback & Skin StimulationShape-Changing Interfaces & Soft Robotic MaterialsImmersion & Presence ResearchCHI
VisCourt: In-Situ Guidance for Interactive Tactic Training in Mixed RealityIn team sports like basketball, understanding and executing tactics---coordinated plans of movements among players---are crucial yet complex, requiring extensive practice. These tactics require players to develop a keen sense of spatial and situational awareness. Traditional coaching methods, which mainly rely on basketball tactic boards and video instruction, often fail to bridge the gap between theoretical learning and the real-world application of tactics, due to shifts in view perspectives and a lack of direct experience with tactical scenarios. To address this challenge, we introduce VisCourt, a Mixed Reality (MR) tactic training system, in collaboration with a professional basketball team. To set up the MR training environment, we employed semi-automatic methods to simulate realistic 3D tactical scenarios and iteratively designed visual in-situ guidance. This approach enables full-body engagement in interactive training sessions on an actual basketball court and provides immediate feedback, significantly enhancing the learning experience. A user study with athletes and enthusiasts shows the effectiveness and satisfaction with VisCourt in basketball training and offers insights for the design of future SportsXR training systems.2024LCLiqi Cheng et al.Full-Body Interaction & Embodied InputMixed Reality WorkspacesImmersion & Presence ResearchUIST
Stick-To-XR: Understanding Stick-Based User Interface Design for Extended RealityThis work explores the design of stick-shaped Tangible User Interfaces (TUI) for Extended Reality (XR). While sticks are widely used in everyday objects, their applications as a TUI in XR have not been systematically studied. We conducted a participatory design session with twelve experts in XR and Human-Computer Interaction to investigate the affordances of stick-based objects and how to utilize them in XR. As a result, we presented a taxonomy of stick-based objects' affordances in terms of their functions and holding gestures. Following that, we proposed four types of stick-based XR controller forms and discussed their advantages and limitations. In the end, we juxtaposed twenty-six existing XR controllers against our proposed forms and identified Landed (Cane) Stick, Thin Stick's flexible usages, and Modular Design as the major opportunities that remain unexamined yet for stick-based XR TUI design.2024YZYaying Zhang et al.Hand Gesture RecognitionMixed Reality WorkspacesDIS
Communication, Collaboration, and Coordination in a Co-located Shared Augmented Reality Game: Perspectives From Deaf and Hard of Hearing PeopleCo-located collaborative shared augmented reality (CS-AR) environments have gained considerable research attention, mainly focusing on design, implementation, accuracy, and usability. Yet, a gap persists in our understanding regarding the accessibility and inclusivity of such environments for diverse user groups, such as deaf and Hard of Hearing (DHH) people. To investigate this domain, we used Urban Legends, a multiplayer game in a co-located CS-AR setting. We conducted a user study followed by one-on-one interviews with 17 DHH participants. Our findings revealed the usage of multimodal communication (verbal and non-verbal) before and during the game, impacting the amount of collaboration among participants and how their coordination with AR components, their surroundings, and other participants improved throughout the rounds. We utilize our data to propose design enhancements, including onscreen visuals and speech-to-text transcription, centered on participant perspectives and our analysis.2024SLSanzida Mojib Luna et al.Rochester Institute of TechnologySocial & Collaborative VRDeaf & Hard-of-Hearing Support (Captions, Sign Language, Vibration)Accessible GamingCHI
Make Interaction Situated: Designing User Acceptable Interaction for Situated Visualization in Public EnvironmentsSituated visualization blends data into the real world to fulfill individuals’ contextual information needs. However, interacting with situated visualization in public environments faces challenges posed by users’ acceptance and contextual constraints. To explore appropriate interaction design, we first conduct a formative study to identify users’ needs for data and interaction. Informed by the findings, we summarize appropriate interaction modalities with eye-based, hand-based and spatially-aware object interaction for situated visualization in public environments. Then, through an iterative design process with six users, we explore and implement interactive techniques for activating and analyzing with situated visualization. To assess the effectiveness and acceptance of these interactions, we integrate them into an AR prototype and conduct a within-subjects study in public scenarios using conventional hand-only interactions as the baseline. The results show that participants preferred our prototype over the baseline, attributing their preference to the interactions being more acceptable, flexible, and practical in public.2024QZQian Zhu et al.The Hong Kong University of Science and Technology, The Hong Kong University of Science and TechnologyAR Navigation & Context AwarenessContext-Aware ComputingField StudiesCHI
AdaptiveVoice: Cognitively Adaptive Voice Interface for Driving AssistanceCurrent voice assistants present messages in a predefined format without considering users’ mental states. This paper presents an optimization-based approach to alleviate this issue which adjusts the level of details and speech speed of the voice messages according to the estimated cognitive load of the user. In the first user study (N=12), we investigated the impact of cognitive load on user performance. The findings reveal significant differences in preferred message formats across five cognitive load levels, substantiating the need for voice message adaptation. We then implemented AdaptiveVoice, an algorithm based on combinatorial optimization to generate adaptive voice messages in real-time. In the second user study (N=30) conducted in a VR-simulated driving environment, we compared AdaptiveVoice with a fixed format baseline, with and without visual guidance on the Heads-up display(HUD). Results indicate that users benefit from AdaptiveVoice with reduced response time and improved driving performance, particularly when it is augmented with HUD.2024SWShaoyue Wen et al.New York University, Xi'an Jiaotong-Liverpool UniversityHead-Up Display (HUD) & Advanced Driver Assistance Systems (ADAS)Voice User Interface (VUI) DesignCHI
"Please Be Nice": Robot Responses to User Bullying - Measuring Performance Across Aggression LevelsAs robots become integral to public services, addressing harmful user behaviors like bullying is crucial. Existing research often overlooks the gradual nature of human bullying. This study fills this gap by exploring how robots can counter bullying through optimized responses. Using a simulated human-robot interaction study, we manipulated robot response behaviors and styles across escalating bullying severity. Results show that empathetic verbal responses promptly reduce users' bullying tendencies by eliciting remorse and redirecting attention to social awareness. However, users' underlying dispositions may override these reflexive reactions, emphasizing the need for a holistic understanding. In conclusion, a comprehensive approach is essential, involving immediate reaction optimization, emotional state assessment, and ongoing behavioral adjustment through empathetic dialogue. By implementing such strategies, we can transform human-robot relationships from potential bullying situations to harmonious interactions. This study provides an empirical foundation for response protocols that discourage bullying and enhance mutual understanding.2024YLYiming Luo et al.Xian UniversitySocial Robot InteractionCHI
Data Cubes in Hand: A Design Space of Tangible Cubes for Visualizing 3D Spatio-Temporal Data in Mixed RealityTangible interfaces in mixed reality (MR) environments allow for intuitive data interactions. Tangible cubes, with their rich interaction affordances, high maneuverability, and stable structure, are particularly well-suited for exploring multi-dimensional data types. However, the design potential of these cubes is underexplored. This study introduces a design space for tangible cubes in MR, focusing on interaction space, visualization space, sizes, and multiplicity. Using spatio-temporal data, we explored the interaction affordances of these cubes in a workshop (N=24). We identified unique interactions like rotating, tapping, and stacking, which are linked to augmented reality (AR) visualization commands. Integrating user-identified interactions, we created a design space for tangible-cube interactions and visualization. A prototype visualizing global health spending with small cubes was developed and evaluated, supporting both individual and combined cube manipulation. This research enhances our grasp of tangible interaction in MR, offering insights for future design and application in diverse data contexts.2024SHShuqi He et al.Xi'an Jiaotong - Liverpool UniversityMixed Reality WorkspacesInteractive Data VisualizationTime-Series & Network Graph VisualizationCHI
Exploration of Foot-based Text Entry Techniques for Virtual Reality EnvironmentsFoot-based input can serve as a supplementary or alternative approach to text entry in virtual reality (VR). This work explores the feasibility and design of foot-based techniques that are hands-free. We first conducted a preliminary study to assess foot-based text entry in standing and seated positions with tap and swipe input approaches. The findings showed that foot-based text input was feasible, with the possibility for performance and usability improvements. We then developed three foot-based techniques, including two tap-based techniques (FeetSymTap and FeetAsymTap) and one swipe-based technique (FeetGestureTap), and evaluated their performance via another user study. The results show that the two tap-based techniques supported entry rates of 11.12 WPM and 10.80 WPM, while the swipe-based technique led to 9.16 WPM. Our findings provide a solid foundation for the future design and implementation of foot-based text entry in VR and have the potential to be extended to MR and AR.2024TWTingjie Wan et al.Xi'an Jiaotong-Liverpool UniversityFoot & Wrist InteractionSocial & Collaborative VRImmersion & Presence ResearchCHI
AR-Enhanced Workouts: Exploring Visual Cues for At-Home Workout Videos in AR EnvironmentIn recent years, with growing health consciousness, at-home workout has become increasingly popular for its convenience and safety. Most people choose to follow video guidance during exercising. However, our preliminary study revealed that fitness-minded people face challenges when watching exercise videos on handheld devices or fixed monitors, such as limited movement comprehension due to static camera angles and insufficient feedback. To address these issues, we reviewed popular workout videos, identified user requirements, and came up with an augmented reality (AR) solution. Following a user-centered iterative design process, we proposed a design space of AR visual cues for workouts and implemented an AR-based application. Specifically, we captured users’ exercise performance with pose-tracking technology and provided feedback via AR visual cues. Two user experiments showed that incorporating AR visual cues could improve movement comprehension and enable users to adjust their movements based on real-time feedback. Finally, we presented several suggestions to inspire future design and apply AR visual cues to sports training.2023YWYingcai Wu et al.Fitness Tracking & Physical Activity MonitoringContext-Aware ComputingUIST
Predicting Gaze-based Target Selection in Augmented Reality Headsets based on Eye and Head Endpoint DistributionsTarget selection is a fundamental task in interactive Augmented Reality (AR) systems. Predicting the intended target of selection in such systems can provide users with a smooth, low-friction interaction experience. Our work aims to predict gaze-based target selection in AR headsets with eye and head endpoint distributions, which describe the probability distribution of eye and head 3D orientation when a user triggers a selection input. We first conducted a user study to collect users’ eye and head behavior in a gaze-based pointing selection task with two confirmation mechanisms (air tap and blinking). Based on the study results, we then built two models: a unimodal model using only eye endpoints and a multimodal model using both eye and head endpoints. Results from a second user study showed that the pointing accuracy is improved by approximately 32% after integrating our models into gaze-based selection techniques.2023YWYushi Wei et al.Xi'an Jiaotong-Liverpool UniversityEye Tracking & Gaze InteractionAR Navigation & Context AwarenessCHI
Tasks of a Different Color: How Crowdsourcing Practices Differ per Complex Task Type and Why This MattersCrowdsourcing in China is a thriving industry. Among its most interesting structures, we find crowdfarms, in which crowdworkers self-organize as small organizations to tackle macrotasks. Little, however, is known as to which practices these crowdfarms use to tackle the macrotasks, and this goes hand in hand with the current practice of the HCI research community to treat all forms of complex crowdsourcing work as practically the same. However, macrotasks differ substantially regarding structure and decomposability. Treating them under one umbrella term - macrotasking - can lead to an imprecise understanding of the workforce involved. We address this gap by examining the work practices of 31 Chinese crowdfarms on the four main macrotask types, namely: modular, interlaced, wicked, and container macrotasks. Our results confirm essential differences in how these nascent crowd organizations address different macrotasks and shed light on what platforms can do to improve the uptake of such work.2023YWYihong Wang et al.Xi'an Jiaotong-Liverpool UniversityMental Health Apps & Online Support CommunitiesCrowdsourcing Task Design & Quality ControlDeveloping Countries & HCI for Development (HCI4D)CHI
DreamVR: Curating an Interactive Exhibition in Social VR Through an Autobiographical Design StudyVirtual exhibitions have long been regarded as an extension of information delivery for physical exhibitions. However, what virtual exhibitions can offer audiences as a novel experience independently from physical exhibitions has been largely unexplored. In this study, we aim to understand the promises and challenges of experiencing and curating exhibitions in VR by interviewing nine expert curators. Drawing from expert insights, we summarized a set of design guidelines to inform what we can learn and adapt from physical exhibitions when curating in VR. Then, using an autobiographical design approach, we curated an interactive exhibition in VRChat to explore novel interaction techniques. We also hosted an open tour guide in the user study to validate our design guidelines with thirty participants. Results show that our approach of curating an exhibition in VRChat provided the participants with engaging and novel experiences interacting with the exhibits and other audiences.2023JCJiaxun Cao et al.Duke Kunshan University, Duke Kunshan UniversitySocial & Collaborative VRInteractive Narrative & Immersive StorytellingCHI
Reflexivity of Account, Professional Vision, and Computer-Supported Cooperative Work: Working in the Maritime DomainComputer-supported cooperative work (CSCW) researchers have plenty to say about designing through texts. However, the relationships between implementation and design are often misunderstood, by engineering project systems developers who design collaborative systems and others. This problem is not new; how to utilize CSCW insights effectively and correctly in engineering projects has long been a concern in the CSCW community. By reviewing year-long, multiple-site ethnographic studies in the maritime domain conducted since 2015, this paper reflects on the “reflexivity of account” and “professional vision” as complementary concepts for ensuring that actors’ actions and reasonings are designed in such a way that actors – in this case, maritime operators, systems developers, educators, policymakers and shipowners – are accountable in and through their member groups. Rather than generalizing my findings and my role in the maritime domain as explicit knowledge to help other CSCW researchers in studying engineering projects, the goal of this paper is to seek intersubjective knowledge of my fieldwork to trigger the utilization of CSCW insights as a fundamental basis for facilitating systems design. The aim is to shape and reshape systems in line with what those actors do in reality.2021YPYushan PanSpecialist and Collaborative Work // Algorithmic FairnessCSCW
iText: Hands-free Text Entry on an Imaginary Keyboard for Augmented Reality SystemsText entry is an important and frequent task in interactive devices including augmented reality head-mounted displays (AR HMDs). In current AR HMDs, there are still two main open challenges to overcome for efficient and usable text entry: arm fatigue due to mid-air input and visual occlusion because of their small see-through displays. To address these challenges, we present iText, a technique for AR HMDs that is hands-free and is based on an imaginary (invisible) keyboard. We first show that it is feasible and practical to use an imaginary keyboard on AR HMDs. Then, we evaluated its performance and usability with three hands-free selection mechanisms: eye blinks (E-Type), dwell (D-Type), and swipe gestures (G-Type). Our results show that users could achieve an average text entry speed of 11.95, 9.03 and 9.84 words per minutes (WPM) with E-Type, D-Type, and G-Type, respectively. Given that iText with E-Type outperformed the other two selection mechanisms in text entry rate and subjective feedback, we ran a third, 5-day study. Our results show that iText with E-Type can achieve an average text entry rate of 13.76 WPM with a mean word error rate of 1.5\%. In short, iText can enable efficient eyes-free text entry and can be useful for various application scenarios in AR HMDs.2021XLXueshi Lu et al.Voice User Interface (VUI) DesignAR Navigation & Context AwarenessUIST
Gaze-Supported 3D Object Manipulation in Virtual RealityThis paper investigates integration, coordination, and transition strategies of gaze and hand input for 3D object manipulation in VR. Specifically, this work aims to understand whether incorporating gaze input can benefit VR object manipulation tasks, and how it should be combined with hand input for improved usability and efficiency. We designed four gaze-supported techniques that leverage different combination strategies for object manipulation and evaluated them in two user studies. Overall, we show that gaze did not offer significant performance benefits for transforming objects in the primary working space, where all objects were located in front of the user and within the arm-reach distance, but can be useful for a larger environment with distant targets. We further offer insights regarding combination strategies of gaze and hand input, and derive implications that can help guide the design of future VR systems that incorporate gaze input for 3D object manipulation.2021DYDifeng Yu et al.The University of MelbourneEye Tracking & Gaze InteractionImmersion & Presence ResearchCHI