Demystifying Reward Design in Reinforcement Learning for Upper Extremity Interaction: Practical Guidelines for Biomechanical Simulations in HCIDesigning effective reward functions is critical for reinforcement learning-based biomechanical simulations, yet HCI researchers and practitioners often waste (computation) time with unintuitive trial-and-error tuning. This paper demystifies reward function design by systematically analyzing the impact of effort minimization, task completion bonuses, and target proximity incentives on typical HCI tasks such as pointing, tracking, and choice reaction. We show that proximity incentives are essential for guiding movement, while completion bonuses ensure task success. Effort terms, though optional, help refine motion regularity when appropriately scaled. We perform an extensive analysis of how sensitive task success and completion time depend on the weights of these three reward components. From these results we derive practical guidelines to create plausible biomechanical simulations without the need for reinforcement learning expertise, which we then validate on remote control and keyboard typing tasks. This paper advances simulation-based interaction design and evaluation in HCI by improving the efficiency and applicability of biomechanical user modeling for real-world interface development.2025HSHannah Selder et al.Human Pose & Activity RecognitionComputational Methods in HCIUIST
Design Activity Simulation: Opportunities and Challenges in Using Multiple Communicative AI Agents to Tackle Design ProblemsLarge Language Models (LLMs) can enhance structured design thinking, yet existing copilot approaches integrate them into human workflows rather than exploring their autonomous potential. This paper investigates how LLM-based communicative AI agents can independently tackle open-ended design problems and how their strengths and limitations inform human-AI collaboration. We iteratively design a system where AI agents play different roles and simulate human design activity through conversational turns. The agents investigate user needs, identify design constraints, and explore the design space, with useful insights emerging from their interactions. To assess reasoning quality, we conducted a human jury evaluation with five HCI researchers and explored potential applications through a contextual inquiry with seven professionals. Our findings demonstrate that integrating human design thinking techniques enhances AI reasoning. AI agents effectively tackle design problems, generating low-novelty yet well-grounded and practical solutions that meet key design requirements.2025BYBoyin Yang et al.Human-LLM CollaborationCreative Collaboration & Feedback SystemsKnowledge Worker Tools & WorkflowsCUI
Exclusion Rates among Disabled and Older Users of Virtual and Augmented RealityThis paper examines the levels of exclusion encountered by disabled and older users of consumer-level VR and AR technology and identifies methods formed by people with diverse access needs to circumvent encountered barriers to use. First, we estimate exclusion rates for a selection of nine immersive experiences of VR and AR, computed using population statistics data for the United Kingdom (UK). We then present an empirical lab-based study evaluating the usability of the same VR and AR experiences. The study involved 60 UK-based participants with varying access needs and the study results were used to calculate the empirical exclusion rates. Both the estimated and empirical exclusion rates display high levels of exclusion, which for the more complex experiences in the study reached 100%. However, multiple participants overcame usability barriers and completed experiences through provided assistance and self-initiated adaptations, suggesting that future VR and AR can become more inclusive if designed to counter these barriers.2025RERosella P. Galindo Esparza et al.Brunel University London, Brunel Design SchoolIdentity & Avatars in XRUniversal & Inclusive DesignCHI
Intimate Data Sharing: Enhancing Transparency and Control in Fertility TrackingFertility trackers are popular for self-monitoring menstrual cycles and managing other aspects of reproductive or sexual health. However, the intimate nature of fertility tracking raises particular concerns about potential data (mis)use. Our study deepens understandings of fertility tracker data sharing and presents co-created mechanisms to enhance user agency over their data in intimate contexts. To achieve this, we first analysed the network transmissions from eight fertility tracker products, observing that many data transmissions appear to be tied to particular uses of the tracker and that the products communicate with endpoints associated with various organisations across different countries. This raises concerns about how intimate data is governed, used, and shared. To understand user attitudes towards data sharing in intimate contexts, we then conducted a survey exploring factors influencing user data sharing preferences. Our findings reveal that users desire transparency and control mechanisms and that their willingness to share data is influenced by contextual factors, including the third parties involved, the purposes of data collection, and the sensitivity of the data. Building on these findings, we worked with users to co-design ten concrete mechanisms for enhancing data transparency and control throughout fertility tracker product usage lifecycles. In all, our mixed-method study provides an in-depth understanding of fertility tracker data flows and preferences and proposes actionable mechanisms designers can utilise to support and protect data rights in intimate data ecosystems.2025AHAnna Ida Hudig et al.University of Cambridge, Department of Computer Science and TechnologyReproductive & Women's HealthPrivacy by Design & User ControlCHI
Making Hardware Devices at Scale is Still Hard: Challenges and Opportunities for the HCI CommunityEmbedded systems and interactive devices form an essential interface between the physical and digital world and are understandably an important focus for the HCI research community. However, scaling an interactive prototype of a new device concept to enable effective evaluation or to support the transition to a production-ready device is incredibly challenging. To better understand the issues innovators face when scaling up interactive device prototypes we report the results from 22 interviews with practitioners in the interactive device field, including eight academics involved in the HCI and manufacturing research communities. In our two-phase analysis we identify and validate the following four recurring themes. First and foremost is the observation that ``creating relationships with industry'' is hard. Second, ``effective communication requires a lot of effort'' despite the availability of modern collaboration tools. Thirdly, we observed that ``understanding the manufacturer's perspective'' can be difficult. Finally, ``prototyping is nothing like production''---the vast difference between these two activities still surprises many. Additionally, our university-based participants gave us further insights and helped us to identify challenges specific to the academic context, pointing to a number of opportunities relating to hardware device scaling.2025BKBo Kang et al.University of CambridgeCircuit Making & Hardware PrototypingCHI
Micro-narratives: A Scalable Method for Eliciting Stories of People’s Lived ExperienceEngaging with people's lived experiences is foundational for HCI research and design. This paper introduces a novel narrative elicitation method to empower people to easily articulate ‘micro-narratives’ emerging from their lived experiences, irrespective of their writing ability or background. Our approach aims to enable at-scale collection of rich, co-created datasets that highlight target populations' voices with minimal participant burden, while precisely addressing specific research questions. To pilot this idea, and test its feasibility, we: (i) developed an AI-powered prototype, which leverages LLM-chaining to scaffold the cognitive steps necessary for users’ narrative articulation; (ii) deployed it in three mixed-methods studies involving over 380 users; and (iii) consulted with established academics as well as C-level staff at (inter)national non-profits to map out potential applications. Both qualitative and quantitative findings show the acceptability and promise of the micro-narrative method, while also identifying the ethical and safeguarding considerations necessary for any at-scale deployments.2025ASAmira Skeggs et al.University of Cambridge, MRC Cognition and Brain Sciences UnitHuman-LLM CollaborationParticipatory DesignUser Research Methods (Interviews, Surveys, Observation)CHI
Copying style, Extracting value: Illustrators’ Perception of AI Style Transfer and its Impact on Creative LaborGenerative text-to-image models are disrupting the lives of creative professionals. Specifically, illustrators are threatened by models that claim to extract and reproduce their style. Yet, research on style transfer has rarely focused on their perspectives. We provided four illustrators with a model fine-tuned to their style and conducted semi-structured interviews about the model’s successes, limitations, and potential uses. Evaluating their output, artists reported that style transfer successfully copies aesthetic fragments but is limited by content-style disentanglement and lacks the crucial emergent quality of their style. They also deemed the others’ copies more successful. Understanding the results of style transfer as “boundary objects,” we analyze how they can simultaneously be considered unsuccessful by artists and poised to replace their work by others. We connect our findings to critical HCI frameworks, demonstrating that style transfer, rather than merely a Creativity Support Tool, should also be understood as a supply chain optimization one.2025JPJulien Porquet et al.University of CambridgeAI Ethics, Fairness & AccountabilityMotor Impairment Assistive Input TechnologiesInclusive DesignCHI
Exploring Multimodal Generative AI for Education through Co-design Workshops with StudentsMultimodal large language models (MLLMs) are Generative AI models that take different modalities such as text, audio, and video as input and generate appropriate multimodal output. Since such models will be integrated into future educational tools, a human-centered design approach that takes students’ perspectives into account is essential while designing such applications. This paper describes two co-design workshops which were conducted with 79 student groups to examine how they design and prototype future educational tools integrated with MLLMs. Through various activities in the workshops, students discussed relevant educational problems, created journey maps, storyboards and low fidelity prototypes for their applications, and evaluated their applications based on relevant design principles. We found that students’ applications used MLLMs for important learning environment design features such as multimodal content creation, personalization, and feedback. Based on these findings, we discuss future research directions for the design of multimodality in generative AI educational applications.2025PPPrajish Prasad et al.FLAME University, School of Computing and Data SciencesGenerative AI (Text, Image, Music, Video)K-12 Digital Education ToolsCHI
SmarTeeth: Augmenting Manual Toothbrushing with In-ear MicrophonesImproper toothbrushing practices persist as a primary cause of oral health issues such as tooth decay and gum disease. Despite the availability of high-end electric toothbrushes that offer some guidance, manual toothbrushes remain widely used due to their simplicity and convenience. We present SmarTeeth, an earable-based toothbrushing monitoring system designed to augment manual toothbrushing with functionalities typically offered only by high-end electric toothbrushes, such as brushing surface tracking. The underlying idea of SmarTeeth is to leverage in-ear microphones on earphones to capture toothbrushing sounds transmitted through the oral cavity to ear canals through facial bones and tissues. The distinct propagation paths of brushing sounds from various dental locations to each ear canal provide the foundational basis for our methods to accurately identify different brushing locations. By extracting customized features from these sounds, we can detect brushing locations using a deep-learning model. With only one registration session (~2 mins) for a new user, the average accuracy is 92.7% for detecting six regions and 75.6% for sixteen tooth surfaces. With three registration sessions (~6 mins), the performance can be boosted to 98.8% and 90.3% for six-region and sixteen-surface tracking, respectively. A key advantage of using earphones for monitoring is that they provide natural auditory feedback to alert users when they are overbrushing or underbrushing. Comprehensive evaluation validates the effectiveness of SmarTeeth under various conditions (different users, brushes, orders, noise, etc.), and the feedback from the user study (N=13) indicates that users found the system highly useful (6.0/7.0) and reported a low workload (2.5/7.0) while using it. Our findings suggest that SmarTeeth could offer a scalable and effective solution to improve oral health globally by providing manual toothbrush users with advanced brushing monitoring capabilities.2025QYQiang Yang et al.University of Cambridge, Department of Computer Science and TechnologyBiosensors & Physiological MonitoringElectronic Textiles (E-textiles)Context-Aware ComputingCHI
AlphaPIG: The Nicest Way to Prolong Interactive Gestures in Extended RealityMid-air gestures serve as a common interaction modality across Extended Reality (XR) applications, enhancing engagement and ownership through intuitive body movements. However, prolonged arm movements induce shoulder fatigue—known as "Gorilla Arm Syndrome"—degrading user experience and reducing interaction duration. Although existing ergonomic techniques derived from Fitts' law (such as reducing target distance, increasing target width, and modifying control-display gain) provide some fatigue mitigation, their implementation in XR applications remains challenging due to the complex balance between user engagement and physical exertion. We present \textit{AlphaPIG}, a meta-technique designed to \textbf{P}rolong \textbf{I}nteractive \textbf{G}estures by leveraging real-time fatigue predictions. AlphaPIG assists designers in extending and improving XR interactions by enabling automated fatigue-based interventions. Through adjustment of intervention timing and intensity decay rate, designers can explore and control the trade-off between fatigue reduction and potential effects such as decreased body ownership. We validated AlphaPIG's effectiveness through a study (N=22) implementing the widely-used Go-Go technique. Results demonstrated that AlphaPIG significantly reduces shoulder fatigue compared to non-adaptive Go-Go, while maintaining comparable perceived body ownership and agency. Based on these findings, we discuss positive and negative perceptions of the intervention. By integrating real-time fatigue prediction with adaptive intervention mechanisms, AlphaPIG constitutes a critical first step towards creating fatigue-aware applications in XR.2025YLZhuying Li et al.Monash UniversityFull-Body Interaction & Embodied InputImmersion & Presence ResearchCHI
Seeing and Touching the Air: Unraveling Eye-Hand Coordination in Mid-Air Gesture Typing for Mixed RealityMid-air text entry in mixed reality (MR) headsets has shown promise but remains less efficient than traditional input methods. While research has focused on improving typing performance, the mechanics of mid-air gesture typing, especially eye-hand coordination, are less understood. This paper investigates visuomotor coordination of mid-air gesture keyboards through a user study (n=16) comparing gesture typing on a tablet and in mid-air. Through an expert task we demonstrate that users were able to achieve a comparable text input performance. Our in-depth analysis of eye-hand coordination reveals significant differences in the eye-hand coordination patterns between gesture typing on a tablet and in-air. The mid-air gesture typing necessitates almost all of the visual attention on the keyboard area and a more consistent synchronization in eye-hand coordination to compensate for the increased motor and cognitive demands without physical boundaries. These insights provide important implications for the design of more efficient text input methods.2025JHJinghui Hu et al.University of Cambridge, Department of EngineeringHand Gesture RecognitionFull-Body Interaction & Embodied InputEye Tracking & Gaze InteractionCHI
BreathPro: Monitoring Breathing Mode during Running with EarablesHu 等人开发 BreathPro 系统，利用耳穿戴设备传感器实时监测跑步时的呼吸模式，为运动健康监测提供新方案。2024CHChangshuo Hu et al.Fitness Tracking & Physical Activity MonitoringBiosensors & Physiological MonitoringUbiComp
SIM2VR: Towards Automated Biomechanical Testing in VRAutomated biomechanical testing has great potential for the development of VR applications, as initial insights into user behaviour can be gained in silico early in the design process. In particular, it allows prediction of user movements and ergonomic variables, such as fatigue, prior to conducting user studies. However, there is a fundamental disconnect between simulators hosting state-of-the-art biomechanical user models and simulators used to develop and run VR applications. Existing user simulators often struggle to capture the intricacies of real-world VR applications, reducing ecological validity of user predictions. In this paper, we introduce SIM2VR, a system that aligns user simulation with a given VR application by establishing a continuous closed loop between the two processes. This, for the first time, enables training simulated users directly in the same VR application that real users interact with. We demonstrate that SIM2VR can predict differences in user performance, ergonomics and strategies in a fast-paced, dynamic arcade game. In order to expand the scope of automated biomechanical testing beyond simple visuomotor tasks, advances in cognitive models and reward function design will be needed.2024FFFlorian Fischer et al.Human Pose & Activity RecognitionVR Medical Training & RehabilitationUIST
Justice-oriented Design Listening: Participatory Ecoacoustics with a Ghanaian Forest CommunityDespite a long tradition of ‘non-expert’ participation in ecoacoustics research, asymmetrical distribution of resources and engagement between the Global North and Global South continue, extending to ecoacoustic sensing and design. Whilst there exists a growing body of work in Participatory Design (PD) addressing the technical and social challenges of ecoacoustic research, we find that popular PD methods inadequately address design justice and decolonising agendas. Through participatory ecoacoustic sensing and design engagements with a forest community in Ghana, we highlight the tensions that emerge when employing visual and written modes of PD in a context where an oral approach to creativity and communication is more appropriate. We present Justice-oriented Design Listening, an acoustically-mediated approach to PD, described through three modes: polyphony, pace and transformation. This work contributes to calls for design justice by developing a methodological approach that facilitates pluralistic participation in design when developing conservation technologies in non-Western contexts.2024JLJoycelyn Longdon et al.University of CambridgeParticipatory DesignSustainable HCIEcological Design & Green ComputingCHI
A Human Information Processing Theory of the Interpretation of Visualizations: Demonstrating Its UtilityProviding an approach to model the memory structures that humans build as they use visualizations could be useful for researchers, designers and educators in the field of information visualization. Cheng and colleagues formulated Representation Interpretive Structure Theory (RIST) for that purpose. RIST adopts a human information processing perspective in order to address the immediate, short timescale, cognitive load likely to be experienced by visualization users. RIST is operationalized in a graphical modeling notation and browser-based editor. This paper demonstrates the utility of RIST by showing that (a): RIST models are compatible with established empirical and computational cognitive findings about differences in human performance on alternative representations; (b) they can encompass existing explanations from the literature; and, (c) they provide new explanations about causes of those performance differences.2024PCPeter Cheng et al.University of SussexVisualization Perception & CognitionComputational Methods in HCICHI
Conceptualising Fatness within HCI: A Call for Fat Liberation Fatness sits at the intersection of many systems of oppression, such as race, gender, class, and (dis)ability. Anti-fat bias happens out in the open and is prevalent in Western society, yet there has been little to no consideration for the wider impact of digital systems in exacerbating, recreating, and repurposing anti-fat bias, or any engagement with designing for fat justice. Therefore, this paper argues that there needs to be a consideration for fat dignity in the design of digital systems, and an investigation of the (un)intended consequences of the datafication of fat lives. This paper offers a scoping literature review of HCI and Fat Studies to identify research gaps and argues that both disciplines would benefit from collaboration. Specifically, the standard of design justice would be increased through radical acceptance, and new questions could be asked to critique how technologies have been leveraged to exercise control over our bodies.2024ASAisha SobeyUniversity of CambridgeGender & Race Issues in HCIEmpowerment of Marginalized GroupsTechnology Ethics & Critical HCICHI
Mind The Gap: Designers and Standards on Algorithmic System Transparency for UsersMany call for algorithmic systems to be more transparent, yet it is often unclear for designers how to do so in practice. Standards are emerging that aim to support designers in building transparent systems, e.g by setting testable transparency levels, but their efficacy in this regard is not yet understood. In this paper, we use the `Standard for Transparency of Autonomous Systems' (IEEE 7001) to explore designers' understanding of algorithmic system transparency, and the degree to which their perspectives align with the standard's recommendations. Our mixed-method study reveals participants consider transparency important, difficult to implement, and welcome support. However, despite IEEE 7001's potential, many did not find its recommendations particularly appropriate. Given the importance and increased attention on transparency, and because standards like this purport to guide system design, our findings reveal the need for `bridging the gap,' through (i) raising designers’ awareness about the importance of algorithmic system transparency, alongside (ii) better engagement between stakeholders (i.e. standards bodies, designers, users). We further identify opportunities towards developing transparency best practices, as means to help drive more responsible systems going forward.2024BSbianca schor et al.University of CambridgeExplainable AI (XAI)Algorithmic Transparency & AuditabilityPrivacy by Design & User ControlCHI
On the Benefits of Image-Schematic Metaphors when Designing Mixed Reality SystemsA Mixed Reality (MR) system encompasses various aspects, such as visualization and spatial registration of user interface elements, user interactions and interaction feedback. Image-schematic metaphors (ISMs) are universal knowledge structures shared by a wide range of users. They hold a theoretical promise of facilitating greater ease of learning and use for interactive systems without costly adaptations. This paper investigates whether image-schematic metaphors (ISMs) can improve user learning, by comparing an existing MR instruction authoring system with or without ISM enhancements. In a user study with 32 participants, we found that the ISM-enhanced system significantly improved task performance, learnability and mental efficiency compared to the baseline. Participants also rated the ISM-enhanced system significantly higher in terms of perspicuity, efficiency, and novelty. These results empirically demonstrate multiple benefits of ISMs when integrated into the design of this MR system and encourage further studies to explore the wider applicability of ISMs in user interface design.2024JLJingyi Li et al.University of CambridgeMixed Reality WorkspacesCHI
"Oh, sorry, I think I interrupted you": Designing Repair Strategies for Robotic Longitudinal Well-being CoachingRobotic well-being coaches have been shown to successfully promote people’s mental well-being. In order to provide successful coaching, a robotic coach should have the capability to repair the mistakes it makes. However, past works investigating robot mistakes are limited to game or task-based, one-off and in-lab studies. This paper presents a 4-phase design process to design repair strategies for robotic longitudinal well-being coaching with the involvement of real-world stakeholders. The design process consists of the following phases: 1) designing repair strategies with a professional well-being coach; 2) undertaking a longitudinal study with the involvement of experienced users (i.e., who had already interacted with a robotic coach before) to investigate the repair strategies defined in (1); 3) conducting a design workshop with the users of the study in (2) to gather their perspectives on the robotic coach’s repair strategies; 4) discussing the results obtained in (2) and (3) with the mental well-being professional to reflect on how to design repair strategies for robotic coaching. Our results show that users have different expectations for a robotic coach than a human coach, which influences how repair strategies should be designed. We show that different repair strategies (e.g., apologizing, explaining, or repairing empathically) are appropriate in different scenarios, and that preferences for repair strategies change during longitudinal, repeated interactions with the robotic coach.2024MAMinja Axelsson et al.Mental Health Apps & Online Support CommunitiesSocial Robot InteractionHRI
Iris: Passive Visible Light Positioning Using Light Spectral Information"We propose a novel Visible Light Positioning (VLP) method, called Iris, that leverages light spectral information (LSI) to localize individuals in a completely passive manner. This means that the user does not need to carry any device, and the existing lighting infrastructure remains unchanged. Our method uses a background subtraction approach to accurately detect changes in ambient LSI caused by human movement. Furthermore, we design a Convolutional Neural Network (CNN) capable of learning and predicting user locations from the LSI change data. To validate our approach, we implemented a prototype of Iris using a commercial-off-the-shelf light spectral sensor and conducted experiments in two typical real-world indoor environments: a 25 m2 one-bedroom apartment and a 13.3m × 8.4m office space. Our results demonstrate that Iris performs effectively in both artificial lighting at night and in highly dynamic natural lighting conditions during the day. Moreover, Iris outperforms the state-of-the-art passive VLP techniques significantly in terms of localization accuracy and the required density of light sensors. To reduce the overhead associated with multi-channel spectral sensing, we develop and validate an algorithm that can minimize the required number of spectral channels for a given environment. Finally, we propose a conditional Generative Adversarial Network (cGAN) that can artificially generate LSI and reduce data collection effort by 50% without sacrificing localization accuracy."https://doi.org/10.1145/36109132023JHYongquan 'Owen' Hu et al.Context-Aware ComputingUbiquitous ComputingUbiComp