Exploring Levels of Control for a Navigation Assistant for Blind TravelersOnly a small percentage of blind and low-vision people use traditional mobility aids such as a cane or a guide dog. Various assistive technologies have been proposed to address the limitations of traditional mobility aids. These devices often give either the user or the device majority of the control. In this work, we explore how varying levels of control affect the users’ sense of agency, trust in the device, confidence, and successful navigation. We present Glide, a novel mobility aid with two modes for control: Glide-directed and User-directed. We employ Glide in a study (N=9) in which blind or low-vision participants used both modes to navigate through an indoor environment. Overall, participants found that Glide was easy to use and learn. Most participants trusted Glide despite its current limitations, and their confidence and performance increased as they continued to use Glide. Users’ control mode preferences varied in different situations; no single mode "won" in all situations.2023VRVinitha Ranganeni et al.Visual Impairment Technologies (Screen Readers, Tactile Graphics, Braille)Augmentative & Alternative Communication (AAC)Context-Aware ComputingHRI
AdHocProx: Sensing Mobile, Ad-Hoc Collaborative Device Formations using Dual Ultra-Wideband RadiosWe present AdHocProx, a system that uses device-relative, inside-out sensing to augment co-located collaboration across multiple devices, without recourse to externally-anchored beacons -- or even reliance on WiFi connectivity. AdHocProx achives this via sensors including dual ultra-wideband (UWB) radios for sensing distance and angle to other devices in dynamic, ad-hoc arrangements; plus capacitive grip to determine where the user's hands hold the device, and to partially correct for the resulting UWB signal attenuation. All spatial sensing and communication takes place via the side-channel capability of the UWB radios, suitable for small-group collaboration across up to four devices (eight UWB radios). Together, these sensors detect proximity and natural, socially meaningful device movements to enable contextual interaction techniques. We find that AdHocProx can obtain 95% accuracy recognizing various ad-hoc device arrangements in an offline evaluation, with participants particularly appreciative of interaction techniques that automatically leverage proximity-awareness and relative orientation amongst multiple devices.2023RLRichard Li et al.University of WashingtonContext-Aware ComputingUbiquitous ComputingCHI
FlowAR: How Different Augmented Reality Visualizations of Online Fitness Videos Support Flow for At-Home Yoga ExercisesOnline fitness video tutorials are an increasingly popular way to stay fit at home without a personal trainer. However, to keep the screen playing the video in view, users typically disrupt their balance and break the motion flow --- two main pillars for the correct execution of yoga poses. While past research partially addressed this problem, these approaches supported only a limited view of the instructor and simple movements. To enable the fluid execution of complex full-body yoga exercises, we propose FlowAR, an augmented reality system for home workouts that shows training video tutorials as always-present virtual static and dynamic overlays around the user. We tested different overlay layouts in a study with 16 participants, using motion capture equipment for baseline performance. Then, we iterated the prototype and tested it in a furnished lab simulating home settings with 12 users. Our results highlight the advantages of different visualizations and the system's general applicability.2023HJHye-Young Jo et al.KAISTAR Navigation & Context AwarenessFitness Tracking & Physical Activity MonitoringCHI
SpinOcchio: Understanding Haptic-Visual Congruency of Skin-Slip in VR with a Dynamic Grip ControllerThis paper's goal is to understand the haptic-visual congruency perception of skin-slip on the fingertips given visual cues in Virtual Reality (VR). We developed SpinOcchio ('Spin' for the spinning mechanism used, 'Occhio' for the Italian word “eye”), a handheld haptic controller capable of rendering the thickness and slipping of a virtual object pinched between two fingers. This is achieved using a mechanism with spinning and pivoting disks that apply a tangential skin-slip movement to the fingertips. With SpinOcchio, we determined the baseline haptic discrimination threshold for skin-slip, and, using these results, we tested how haptic realism of motion and thickness is perceived with varying visual cues in VR. Surprisingly, the results show that in all cases, visual cues dominate over haptic perception. Based on these results, we suggest applications that leverage skin-slip and grip interaction, contributing further to realistic experiences in VR.2022MKMyung Jin Kim et al.KAISTHaptic WearablesBrain-Computer Interface (BCI) & NeurofeedbackCHI
HapticBots: Distributed Encountered-type Haptics for VR with Multiple Shape-changing Mobile RobotsHapticBots introduces a novel encountered-type haptic approach for Virtual Reality (VR) based on multiple tabletop-size shape-changing robots. These robots move on a tabletop and change their height and orientation to haptically render various surfaces and objects on-demand. Compared to previous encountered-type haptic approaches like shape displays or robotic arms, our proposed approach has an advantage in deployability, scalability, and generalizability---these robots can be easily deployed due to their compact form factor. They can support multiple concurrent touch points in a large area thanks to the distributed nature of the robots. We propose and evaluate a novel set of interactions enabled by these robots which include: 1) rendering haptics for VR objects by providing just-in-time touch-points on the user's hand, 2) simulating continuous surfaces with the concurrent height and position change, and 3) enabling the user to pick up and move VR objects through graspable proxy objects. Finally, we demonstrate HapticBots with various applications, including remote collaboration, education and training, design and 3D modeling, and gaming and entertainment.2021RSRyo Suzuki et al.Mid-Air Haptics (Ultrasonic)Mixed Reality WorkspacesImmersion & Presence ResearchUIST
X-Rings: A Hand-mounted 360 Degree Shape Display for Grasping in Virtual RealityX-Rings is a novel hand-mounted 360 degree shape display for Virtual Reality that renders objects in 3D and responds to user-applied touch and grasping force. Designed as a modular stack of motor-driven expandable rings (5.7-7.7 cm diameter), X-Rings renders radially-symmetric surfaces graspable by the user's whole hand. The device is strapped to the palm, allowing the fingers to freely make and break contact with the device. Capacitance sensors and motor current sensing provide estimates of finger touch states and gripping force. We present the results of a user study evaluating participants’ ability to associate device-rendered shapes with visually-rendered objects as well as a demo application that allows users to freely interact with a variety of objects in a virtual environment.2021EGEric J. Gonzalez et al.Shape-Changing Interfaces & Soft Robotic MaterialsIdentity & Avatars in XRUIST
A Taxonomy of Sounds in Virtual RealityVirtual reality (VR) leverages human sight, hearing and touch senses to convey virtual experiences. For d/Deaf and hard of hearing (DHH) people, information conveyed through sound may not be accessible. To help with future design of accessible VR sound representations for DHH users, this paper contributes a consistent language and structure for representing sounds in VR. Using two studies, we report on the design and evaluation of a novel taxonomy for VR sounds. Study 1 included interviews with 10 VR sound designers to develop our taxonomy along two dimensions: sound source and intent. To evaluate this taxonomy, we conducted another study (Study 2) where eight HCI researchers used our taxonomy to document sounds in 33 VR apps. We found that our taxonomy was able to successfully categorize nearly all sounds (265/267) in these apps. We also uncovered additional insights for designing accessible visual and haptic-based sound substitutes for DHH users.2021DJDhruv Jain et al.Social & Collaborative VRImmersion & Presence ResearchDeaf & Hard-of-Hearing Support (Captions, Sign Language, Vibration)DIS
GamesBond: Bimanual Haptic Illusion of Physically Connected Objects for Immersive VR Using Grip DeformationVirtual Reality experiences, such as games and simulations, typically support the usage of bimanual controllers to interact with virtual objects. To recreate the haptic sensation of holding objects of various shapes and behaviors with both hands, previous researchers have used mechanical linkages between the controllers that render adjustable stiffness. However, the linkage cannot quickly adapt to simulate dynamic objects, nor it can be removed to support free movements. This paper introduces GamesBond, a pair of 4-DoF controllers without physical linkage but capable to create the illusion of being connected as a single device, forming a virtual bond. The two controllers work together by dynamically displaying and physically rendering deformations of hand grips, and so allowing users to perceive a single connected object between the hands, such as a jumping rope. With a user study and various applications we show that GamesBond increases the realism, immersion, and enjoyment of bimanual interaction.2021NRNeung Ryu et al.KAISTIn-Vehicle Haptic, Audio & Multimodal FeedbackShape-Changing Interfaces & Soft Robotic MaterialsFull-Body Interaction & Embodied InputCHI
Haptic PIVOT: On-Demand Handhelds in VRWe present PIVOT, a wrist-worn haptic device that renders virtual objects into the user’s hand on demand. Its simple design comprises a single actuated joint that pivots a haptic handle into and out of the user’s hand, rendering the haptic sensations of grasping, catching, or throwing an object – anywhere in space. Unlike existing hand-held haptic devices and haptic gloves, PIVOT leaves the user’s palm free when not in use, allowing users to make unencumbered use of their hand. PIVOT also enables rendering forces acting on the held virtual objects, such as gravity, inertia, or air-drag, by actively driving its motor while the user is firmly holding the handle. When wearing a PIVOT device on both hands, they can add haptic feedback to bimanual interaction, such as lifting larger objects. In our user study, participants (n=12) evaluated the realism of grabbing and releasing objects of different shape and size with mean score 5.19 on a scale from 1 to 7, rated the ability to catch and throw balls in different directions with different velocities (mean=5.5), and verified the ability to render the comparative weight of held objects with 87% accuracy for ~100g increments.2020RKRobert Kovacs et al.Force Feedback & Pseudo-Haptic WeightHaptic WearablesUIST
Virtual Reality Without Vision: A Haptic and Auditory White Cane to Navigate Complex Virtual WorldsCurrent Virtual Reality (VR) technologies focus on rendering visuospatial effects, and thus are inaccessible for blind or low vision users. We examine the use of a novel white cane controller that enables navigation without vision of large virtual environments with complex architecture, such as winding paths and occluding walls and doors. The cane controller employs a lightweight three-axis brake mechanism to provide large-scale shape of virtual objects. The multiple degrees-of-freedom enables users to adapt the controller to their preferred techniques and grip. In addition, surface textures are rendered with a voice coil actuator based on contact vibrations; and spatialized audio is determined based on the progression of sound through the geometry around the user. We design a scavenger hunt game that demonstrates how our device enables blind users to navigate a complex virtual environment. Seven out of eight users were able to successfully navigate the virtual room (6x6m) to locate targets while avoiding collisions. We conclude with design consideration on creating immersive non-visual VR experiences based on user preferences for cane techniques, and cane material properties.2020ASAlexa F. Siu et al.Stanford UniversitySocial & Collaborative VRVisual Impairment Technologies (Screen Readers, Tactile Graphics, Braille)CHI
CapstanCrunch: A Haptic VR Controller with User-supplied Force FeedbackWe introduce CapstanCrunch, a force resisting, ungrounded haptic controller that renders haptic feedback for touching and grasping both rigid and compliant objects in a VR environment. In contrast to previous controllers, CapstanCrunch renders human-scale forces without the use of large, high force, electrically power consumptive and expensive actuators. Instead, CapstanCrunch integrates a friction-based capstan-plus-cord variable-resistance brake mechanism that is dynamically controlled by a small internal motor. The capstan mechanism magnifies the motor’s force by a factor of around 40. Compared to active force control devices, it is low cost, low electrical power, robust, safe, fast and quiet, while providing high force control to user interaction. We describe the design and implementation of CapstanCrunch and demonstrate its use in a series of VR scenarios. Finally, we evaluate the performance of CapstanCrunch in two user studies and compare our controller with an active haptic controller with the ability to simulate different levels of convincing object rigidity and/or compliance.2019MSMike Sinclair et al.Force Feedback & Pseudo-Haptic WeightFull-Body Interaction & Embodied InputUIST
TORC: A Virtual Reality Controller for In-Hand High-Dexterity Finger InteractionRecent hand-held controllers have explored a variety of haptic feedback sensations for users in virtual reality by producing both kinesthetic and cutaneous feedback from virtual objects. These controllers are grounded to the user's hand and can only manipulate objects through arm and wrist motions, not using the dexterity of their fingers as they would in real life. In this paper, we present TORC, a rigid haptic controller that renders virtual object characteristics and behaviors such as texture and compliance. Users hold and squeeze TORC using their thumb and two fingers and interact with virtual objects by sliding their thumb on TORC's trackpad. During the interaction, vibrotactile motors produce sensations to each finger that represent the haptic feel of squeezing, shearing or turning an object. Our evaluation showed that using TORC, participants could manipulate virtual objects more precisely (e.g., position and rotate objects in 3D) than when using a conventional VR controller.2019JLJaeyeon Lee et al.Microsoft Research & Korea Advanced Institute of Science and TechnologyVibrotactile Feedback & Skin StimulationForce Feedback & Pseudo-Haptic WeightCHI
Haptic Links: Bimanual Haptics for Virtual Reality Using Variable Stiffness ActuationWe present Haptic Links, electro-mechanically actuated physical connections capable of rendering variable stiffness between two commodity handheld virtual reality (VR) controllers. When attached, Haptic Links can dynamically alter the forces perceived between the user’s hands to support the haptic rendering of a variety of two-handed objects and interactions. They can rigidly lock controllers in an arbitrary configuration, constrain specific degrees of freedom or directions of motion, and dynamically set stiffness along a continuous range. We demonstrate and compare three prototype Haptic Links: Chain, Layer-Hinge, and Ratchet-Hinge. We then describe interaction techniques and scenarios leveraging the capabilities of each. Our user evaluation results confirm that users can perceive many two-handed objects or interactions as more realistic with Haptic Links than with typical unlinked VR controllers.2018ESEvan Strasnick et al.Stanford UniversityForce Feedback & Pseudo-Haptic WeightFull-Body Interaction & Embodied InputImmersion & Presence ResearchCHI
Enabling People with Visual Impairments to Navigate Virtual Reality with a Haptic and Auditory Cane SimulationTraditional virtual reality (VR) mainly focuses on visual feedback, which is not accessible for people with visual impairments. We created Canetroller, a haptic cane controller that simulates white cane interactions, enabling people with visual impairments to navigate a virtual environment by transferring their cane skills into the virtual world. Canetroller provides three types of feedback: (1) physical resistance generated by a wearable programmable brake mechanism that physically impedes the controller when the virtual cane comes in contact with a virtual object; (2) vibrotactile feedback that simulates the vibrations when a cane hits an object or touches and drags across various surfaces; and (3) spatial 3D auditory feedback simulating the sound of real-world cane interactions. We designed indoor and outdoor VR scenes to evaluate the effectiveness of our controller. Our study showed that Canetroller was a promising tool that enabled visually impaired participants to navigate different virtual spaces. We discuss potential applications supported by Canetroller ranging from entertainment to mobility training.2018YZYuhang Zhao et al.Microsoft Research, Cornell TechVisual Impairment Technologies (Screen Readers, Tactile Graphics, Braille)CHI
CLAW: A Multifunctional Handheld Haptic Controller for Grasping, Touching, and Triggering in Virtual RealityCLAW is a handheld virtual reality controller that augments the typical controller functionality with force feedback and actuated movement to the index finger. Our controller enables three distinct interactions (grasping virtual object, touching virtual surfaces, and triggering) and changes its corresponding haptic rendering by sensing the differences in the user's grasp. A servo motor coupled with a force sensor renders controllable forces to the index finger during grasping and touching. Using position tracking, a voice coil actuator at the index fingertip generates vibrations for various textures synchronized with finger movement. CLAW also supports a haptic force feedback in the trigger mode when the user holds a gun. We describe the design considerations for CLAW and evaluate its performance through two user studies. The first study obtained qualitative user feedback on the naturalness, effectiveness, and comfort when using the device. The second study investigated the ease of the transition between grasping and touching when using our device.2018ICInrak Choi et al.Stanford UniversityForce Feedback & Pseudo-Haptic WeightHaptic WearablesCHI
Haptic Revolver: Touch, Shear, Texture, and Shape Rendering on a Reconfigurable Virtual Reality ControllerWe present Haptic Revolver, a handheld virtual reality controller that renders fingertip haptics when interacting with virtual surfaces. Haptic Revolver's core haptic element is an actuated wheel that raises and lowers underneath the finger to render contact with a virtual surface. As the user's finger moves along the surface of an object, the controller spins the wheel to render shear forces and motion under the fingertip. The wheel is interchangeable and can contain physical textures, shapes, edges, or active elements to provide different sensations to the user. Because the controller is spatially tracked, these physical features can be spatially registered with the geometry of the virtual environment and rendered on-demand. We evaluated Haptic Revolver in two studies to understand how wheel speed and direction impact perceived realism. We also report qualitative feedback from users who explored three application scenarios with our controller.2018EWEric Whitmire et al.University of WashingtonVibrotactile Feedback & Skin StimulationHaptic WearablesCHI