Less or More: Towards Glanceable Explanations for LLM Recommendations Using Ultra-Small DevicesLarge Language Models (LLMs) have shown remarkable potential in recommending everyday actions as personal AI assistants, while Explainable AI (XAI) techniques are being increasingly utilized to help users understand why a recommendation is given. Personal AI assistants today are often located on ultra-small devices such as smartwatches, which have limited screen space. The verbosity of LLM-generated explanations, however, makes it challenging to deliver glanceable LLM explanations on such ultra-small devices. To address this, we explored 1) spatially structuring an LLM’s explanation text using defined contextual components during prompting and 2) presenting temporally adaptive explanations to users based on confidence levels. We conducted a user study to understand how these approaches impacted user experiences when interacting with LLM recommendations and explanations on ultra-small devices. The results showed that structured explanations reduced users’ time to action and cognitive load when reading an explanation. Always-on structured explanations increased users’ acceptance of AI recommendations. However, users were less satisfied with structured explanations compared to unstructured ones due to their lack of sufficient, readable details. Additionally, adaptively presenting structured explanations was less effective at improving user perceptions of the AI compared to the always-on structured explanations. Together with users' interview feedback, the results led to design implications to be mindful of when personalizing the content and timing of LLM explanations that are displayed on ultra-small devices.2025XWXinru Wang et al.Human-LLM CollaborationExplainable AI (XAI)IUI
SoundScroll: Robust Finger Slide Detection Using Friction Sound and Wrist-Worn MicrophonesSmartwatches have firmly established themselves as a popular wearable form factor. The potential expansion of their interaction space to nearby surfaces offers a promising avenue for enhancing input accuracy and usability beyond the confines of a small screen. However, a key challenge is in detecting continuous contact states with the surface to inform the start and end of stateful interactions. In this paper, we introduce SoundScroll, enabling a rapid and precise determination of contact state and fingertip speed of sliding finger. We leverage vibrations from friction between a moving finger and a surface. Our proof-of-concept wristband captures a dual-channel vibration signal for robust sensing, considering both on-skin and in-air components. Our software predicts a finger sliding state as fast as 20 ms with an accuracy of 93.3%. Augmenting prior approaches detecting tap events, SoundScroll can be a robust, low-latency, and precise contact and motion sensing technique.2024DKDaehwa Kim et al.Vibrotactile Feedback & Skin StimulationFoot & Wrist InteractionSmartwatches & Fitness BandsUbiComp
picoRing: battery-free rings for subtle thumb-to-index inputSmart rings for subtle, reliable finger input offer an attractive path for ubiquitous interaction with wearable computing platforms. However, compared to ordinary rings worn for cultural or fashion reasons, smart rings are much bulkier and less comfortable, largely due to the space required for a battery, which also limits the space available for sensors. This paper presents picoRing, a flexible sensing architecture that enables a variety of battery-free smart rings paired with a wristband. By inductively connecting a wristband-based sensitive reader coil with a ring-based fully-passive sensor coil, picoRing enables the wristband to stably detect the passive response from the ring via a weak inductive coupling. We demonstrate four different rings that support thumb-to-finger interactions like pressing, sliding, or scrolling. When users perform these interactions, the corresponding ring converts each input into a unique passive response through a network of passive switches. Combining the coil-based sensitive readout with the fully-passive ring design enables a tiny ring that weighs as little as 1.5 g and achieves a 13 cm stable readout despite finger bending, and proximity to metal.2024RTRyo Takahashi et al.Force Feedback & Pseudo-Haptic WeightHaptic WearablesFoot & Wrist InteractionUIST
MineXR: Mining Personalized Extended Reality InterfacesExtended Reality (XR) interfaces offer engaging user experiences, but their effective design requires a nuanced understanding of user behavior and preferences. This knowledge is challenging to obtain without the widespread adoption of XR devices. We introduce MineXR, a design mining workflow and data analysis platform for collecting and analyzing personalized XR user interaction and experience data. MineXR enables elicitation of personalized interfaces from participants of a data collection: for any particular context, participants create interface elements using application screenshots from their own smartphone, place them in the environment, and simultaneously preview the resulting XR layout on a headset. Using MineXR, we contribute a dataset of personalized XR interfaces collected from 31 participants, consisting of 695 XR widgets created from 178 unique applications. We provide insights for XR widget functionalities, categories, clusters, UI element types, and placement. Our open-source tools and data support researchers and designers in developing future XR interfaces.2024HCHyunsung Cho et al.Carnegie Mellon UniversityMixed Reality WorkspacesImmersion & Presence ResearchInteractive Data VisualizationCHI
A Meta-Bayesian Approach for Rapid Online Parametric Optimization for Wrist-based InteractionsWrist-based input often requires tuning parameter settings in correspondence to between-user and between-session differences, such as variations in hand anatomy, wearing position, posture, etc. Traditionally, users either work with predefined parameter values not optimized for individuals or undergo time-consuming calibration processes. We propose an online Bayesian Optimization (BO)-based method for rapidly determining the user-specific optimal settings of wrist-based pointing. Specifically, we develop a meta-Bayesian optimization (meta-BO) method, differing from traditional human-in-the-loop BO: By incorporating meta-learning of prior optimization data from a user population with BO, meta-BO enables rapid calibration of parameters for new users with a handful of trials. We evaluate our method with two representative and distinct wrist-based interactions: absolute and relative pointing. On a weighted-sum metric that consists of completion time, aiming error, and trajectory quality, meta-BO improves absolute pointing performance by 22.92% and 21.35% compared to BO and manual calibration, and improves relative pointing performance by 25.43% and 13.60%.2024YLYi-Chi Liao et al.Aalto UniversityVibrotactile Feedback & Skin StimulationForce Feedback & Pseudo-Haptic WeightFoot & Wrist InteractionCHI
Investigating Wrist Deflection Scrolling Techniques for Extended RealityScrolling in extended reality (XR) is currently performed using handheld controllers or vision-based arm-in-front gestures, which have the limitations of encumbering the user's hands or requiring a specific arm posture, respectively. To address these limitations, we investigate freehand, posture-independent scrolling driven by wrist deflection. We propose two novel techniques: Wrist Joystick, which uses rate control, and Wrist Drag, which uses position control. In an empirical study of a rapid item acquisition task and a casual browsing task, both Wrist Drag and Wrist Joystick performed on par with a comparable state-of-the-art technique on one of the two tasks. Further, using a relaxed arm-at-side posture, participants retained their arm-in-front performance for both wrist techniques. Finally, we analyze behavioral and ergonomic data to provide design insights for wrist deflection scrolling. Our results demonstrate that wrist deflection provides a promising method for performant scrolling controls while offering additional benefits over existing XR interaction techniques.2023JFJacqui Fashimpaur et al.Meta Inc.Foot & Wrist InteractionMixed Reality WorkspacesImmersion & Presence ResearchCHI
Investigating Eyes-away Mid-air Typing in Virtual Reality using Squeeze haptics-based Postural ReinforcementIn this paper, we investigate postural reinforcement haptics for mid-air typing using squeeze actuation on the wrist. We propose and validate eye-tracking based objective metrics that capture the impact of haptics on the user's experience, which traditional performance metrics like speed and accuracy are not able to capture. To this end, we design four wrist-based haptic feedback conditions: no haptics, vibrations on keypress, squeeze+vibrations on keypress, and squeeze posture reinforcement + vibrations on keypress. We conduct a text input study with 48 participants to compare the four conditions on typing and gaze metrics. Our results show that for expert qwerty users, posture reinforcement haptics significantly benefit typing by reducing the visual attention on the keyboard by up to 44% relative to no haptics, thus enabling eyes-away behaviors.2023AGAakar Gupta et al.Meta IncVibrotactile Feedback & Skin StimulationHand Gesture RecognitionEye Tracking & Gaze InteractionCHI
XAIR: A Framework of Explainable AI in Augmented RealityExplainable AI (XAI) has established itself as an important component of AI-driven interactive systems. With Augmented Reality (AR) becoming more integrated in daily lives, the role of XAI also becomes essential in AR because end-users will frequently interact with intelligent services. However, it is unclear how to design effective XAI experiences for AR. We propose XAIR, a design framework that addresses when, what, and how to provide explanations of AI output in AR. The framework was based on a multi-disciplinary literature review of XAI and HCI research, a large-scale survey probing 500+ end-users’ preferences for AR-based explanations, and three workshops with 12 experts collecting their insights about XAI design in AR. XAIR's utility and effectiveness was verified via a study with 10 designers and another study with 12 end-users. XAIR can provide guidelines for designers, inspiring them to identify new design opportunities and achieve effective XAI designs in AR.2023XXXuhai Xu et al.Reality Labs Research, University of WashingtonAR Navigation & Context AwarenessExplainable AI (XAI)CHI
RIDS: Implicit Detection of a Selection Gesture Using Hand Motion Dynamics During Freehand Pointing in Virtual RealityFreehand interactions with augmented and virtual reality are grow- ing in popularity, but they lack reliability and robustness. Implicit behavior from users, such as hand or gaze movements, might pro- vide additional signals to improve the reliability of input. In this paper, the primary goal is to improve the detection of a selection gesture in VR during point-and-click interaction. Thus, we propose and investigate the use of information contained within the hand motion dynamics that precede a selection gesture. We built two models that classified if a user is likely to perform a selection gesture at the current moment in time. We collected data during a pointing-and-selection task from 15 participants and trained two models with different architectures, i.e., a logistic regression classifier was trained using predefined hand motion features and a temporal convolutional network (TCN) classifier was trained using raw hand motion data. Leave-one-subject-out cross-validation PR- AUCs of 0.36 and 0.90 were obtained for each model respectively, demonstrating that the models performed well above chance (=0.13). The TCN model was found to improve the precision of a noisy selection gesture by 11.2% without sacrificing recall performance. An initial analysis of the generalizability of the models demonstrated above-chance performance, suggesting that this approach could be scaled to other interaction tasks in the future.2022ZHZhenhong Hu et al.Hand Gesture RecognitionHuman Pose & Activity RecognitionUIST
Optimizing the Timing of Intelligent Suggestion in Virtual RealityIntelligent suggestion techniques can enable low-friction selection-based input within virtual or augmented reality (VR/AR) systems. Such techniques leverage probability estimates from a target prediction model to provide users with an easy-to-use method to select the most probable target in an environment. For example, a system could highlight the predicted target and enable a user to select it with a simple click. However, as the probability estimates can be made at any time, it is unclear when an intelligent suggestion should be presented. Earlier suggestions could save a user time and effort but be less accurate. Later suggestions, on the other hand, could be more accurate but save less time and effort. This paper thus proposes a computational framework that can be used to determine the optimal timing of intelligent suggestions based on user-centric costs and benefits. A series of studies demonstrated the value of the framework for minimizing task completion time and maximizing suggestion usage and showed that it was both theoretically and empirically effective at determining the optimal timing for intelligent suggestions.2022DYDifeng Yu et al.Social & Collaborative VRAI-Assisted Decision-Making & AutomationUIST
False Positives vs. False Negatives: The Effects of Recovery Time and Cognitive Costs on Input Error PreferenceExisting approaches to trading off false positive versus false negative errors in input recognition are based on imprecise ideas of how these errors affect user experience that are unlikely to hold for all situations. To inform dynamic approaches to setting such a tradeoff, two user studies were conducted on how relative preference for false positive versus false negative errors is influenced by differences in the temporal cost of error recovery, and high-level task factors (time pressure, multi-tasking). Participants completed a tile selection task in which false positive and false negative errors were injected at a fixed rate, and the temporal cost to recover from each of the two types of error was varied, and then indicated a preference for one error type or the other, and a frustration rating for the task. Responses indicate that the temporal costs of error recovery can drive both frustration and relative error type preference, and that participants exhibit a bias against false positive errors, equivalent to ~1.5 seconds or more of added temporal recovery time. Several explanations for this bias were revealed, including that false positive errors impose a greater attentional demand on the user, and that recovering from false positive errors imposes a task switching cost.2021BLBen Lafreniere et al.Visualization Perception & CognitionUser Research Methods (Interviews, Surveys, Observation)UIST
Armstrong: An Empirical Examination of Pointing at Non-Dominant Arm-Anchored UIs in Virtual RealityIn virtual reality (VR) environments, asymmetric bimanual interaction techniques can increase users' input bandwidth by complementing their perceptual and motor systems (e.g., using the dominant hand to select 3D UI controls anchored around the non-dominant arm). However, it is unclear how to optimize the layout of such 3D UI controls for near-body and mid-air interactions. We evaluate the performance and limitations of non-dominant arm-anchored 3D UIs in VR environments through a bimanual pointing study. Results demonstrated that targets appearing closer to the skin, located around the wrist, or placed on the medial side of the forearm could be selected more quickly than targets farther away from the skin, located around the elbow, or on the lateral side of the forearm. Based on these results, we developed Armstrong guidelines, demonstrated through a Unity plugin to enable designers to create performance-optimized arm-anchored 3D UI layouts.2021ZLZhen Li et al.University of Toronto, Chatham LabsMid-Air Haptics (Ultrasonic)Haptic WearablesFull-Body Interaction & Embodied InputCHI
Understanding, Detecting and Mitigating the Effects of Coactivations in Ten-Finger Mid-Air Typing in Virtual RealityTyping with ten fingers on a virtual keyboard in virtual or augmented reality exposes a challenging input interpretation problem. There are many sources of noise in this interaction context and these exacerbate the challenge of accurately translating human actions into text. A particularly challenging input noise source arises from the physiology of the hand. Intentional finger movements can produce unintentional coactivations in other fingers. On a physical keyboard, the resistance of the keys alleviates this issue. On a virtual keyboard, coactivations are likely to introduce spurious input events under a naïve solution to input detection. In this paper we examine the features that discriminate intentional activations from coactivations. Based on this analysis, we demonstrate three alternative coactivation detection strategies with high discrimination power. Finally, we integrate coactivation detection into a probabilistic decoder and demonstrate its ability to further reduce uncorrected character error rates by approximately 10% relative and 0.9% absolute.2021CFConor R Foy et al.University of CambridgeHand Gesture RecognitionFull-Body Interaction & Embodied InputEye Tracking & Gaze InteractionCHI
Acustico: Surface Tap Detection and Localization using Wrist-based Acoustic TDOA SensingIn this paper, we present Acustico, a passive acoustic sensing approach that enables tap detection and 2D tap localization on uninstrumented surfaces using a wristworn device. Our technique uses a novel application of acoustic time differences of arrival (TDOA) analysis. We adopt a sensor fusion approach by taking both “surface waves” (i.e., vibrations through surface) and “sound waves” (i.e., vibrations through air) into analysis to improve sensing resolution. We carefully design a sensor configuration to meet the constraints of a wristband form factor. We built a wristband prototype with four acoustic sensors, two accelerometers and two microphones. Through a 20-participant study, we evaluated the performance of our proposed sensing technique for tap detection and localization. Results show that our system reliably detects taps with an F1-score of 0.9987 across different environmental noises and yields high localization accuracies with root-mean-square-errors of 7.6mm (Xaxis) and 4.6mm (Y-axis) across different surfaces and tapping techniques.2020JGJun Gong et al.Foot & Wrist InteractionBiosensors & Physiological MonitoringUIST
Head-Coupled Kinematic Template Matching: A Prediction Model for Ray Pointing in VRThis paper presents a new technique to predict the ray pointer landing position for selection movements in virtual reality (VR) environments. The technique adapts and extends a prior 2D kinematic template matching method to VR environments where ray pointers are used for selection. It builds on the insight that the kinematics of a controller and Head-Mounted Display (HMD) can be used to predict the ray's final landing position and angle. An initial study provides evidence that the motion of the head is a key input channel for improving prediction models. A second study validates this technique across a continuous range of distances, angles, and target sizes. On average, the technique's predictions were within 7.3° of the true landing position when 50% of the way through the movement and within 3.4° when 90%. Furthermore, compared to a direct extension of Kinematic Template Matching, which only uses controller movement, this head-coupled approach increases prediction accuracy by a factor of 1.8x when 40% of the way through the movement.2020RHRorik Henrikson et al.Chatham LabsEye Tracking & Gaze InteractionHuman Pose & Activity RecognitionImmersion & Presence ResearchCHI
Learning Cooperative Personalized Policies from Gaze DataAn ideal Mixed Reality (MR) system would only present virtual information (e.g., a label) when it is useful to the person. However, figuring out when a label is useful is challenging; it depends on a variety of factors, including the current task, previous knowledge, context, etc. In this paper, we propose a Reinforcement Learning (RL) method to learn when to show or hide an object’s label given eye movement data. We demonstrate the capabilities of this approach by showing that an intelligent agent can learn cooperative policies that better support users in a visual search task than design heuristics. Furthermore, we show the applicability of our approach in realistic environments and use cases (e.g., grocery shopping). By posing MR object labeling as an RL control problem we can learn policies implicitly by observing users’ behavior without requiring experience sampling or any other form of supervision.2019CGChristoph Gebhardt et al.Eye Tracking & Gaze InteractionMixed Reality WorkspacesAI-Assisted Decision-Making & AutomationUIST
True Touch: Precise Touch Detection for On-Skin AR/VR InterfacesContemporary AR/VR systems use in-air gestures or handheld controllers for interactivity. This overlooks the skin as a convenient surface for tactile, touch-driven interactions, which are generally more accurate and comfortable than free space interactions. We developed RFTouch, an electrical method that enables very precise touch segmentation by using the body as an RF waveguide. We combine this method with computer vision, enabling a system with both high tracking precision and robust touch detection. Our system requires no cumbersome instrumentation of the fingers or hands, requiring only a single wristband and sensors integrated into the headset. We quantify the accuracy of our approach through a user study and demonstrate how it can enable touchscreen-like interactions on the skin.2019YZYang Zhang et al.Hand Gesture RecognitionImmersion & Presence ResearchOn-Skin Display & On-Skin InputUIST
Pseudo-Haptic Weight: Changing the Perceived Weight of Virtual Objects By Manipulating Control-Display RatioIn virtual reality, the lack of kinesthetic feedback often prevents users from experiencing the weight of virtual objects. Control-to-display (C/D) ratio manipulation has been proposed as a method to induce weight perception without kinesthetic feedback. Based on the fact that lighter (heavier) objects are easier (harder) to move, this method induces an illusory perception of weight by manipulating the rendered position of users' hands---increasing or decreasing their displayed movements. In a series of experiments we demonstrate that C/D-ratio induces a genuine perception of weight, while preserving ownership over the virtual hand. This means that such a manipulation can be easily introduced in current VR experiences without disrupting the sense of presence. We discuss these findings in terms of estimation of physical work needed to lift an object. Our findings provide the first quantification of the range of C/D-ratio that can be used to simulate weight in virtual reality.2019MSMajed Samad et al.Facebook Reality LabsForce Feedback & Pseudo-Haptic WeightCHI