Generative Muscle Stimulation: Providing Users with Physical Assistance by Constraining Multimodal-AI with Embodied KnowledgeElectrical-muscle-stimulation (EMS) can support physical-assistance (e.g., shaking a spray-can before painting). However, EMS-assistance is highly-specialized because it is (1) fixed (e.g., one program for shaking spray-cans, another for opening windows); and (2) non-contextual (e.g., a spray-can for cooking dispenses cooking-oil, not paint—shaking it is unnecessary). Instead, we explore a different approach where muscle-stimulation instructions are generated considering the user’s context (e.g., pose, location, surroundings). The resulting system is more general—enabling unprecedented EMS-interactions (e.g., opening a pill-bottle) yet also replicating existing systems (e.g., Affordance++) without task-specific programming. It uses computer-vision/large-language-models to generate EMS-instructions, constraining these to a muscle-stimulation knowledge-base & joint-limits. In our user-study, we found participants successfully completed physical-tasks while guided by generative-EMS, even when EMS-instructions were (purposely) erroneous. Participants understood generated-gestures and, even during forced-errors, understood partial-instructions, identified errors, and re-prompted the system. We believe our concept marks a shift toward more general-purpose EMS-interfaces.2026YHYun Ho et al.University of ChicagoElectrical Muscle Stimulation (EMS)Generative AI (Text, Image, Music, Video)Human-LLM CollaborationCHI
Seeing with the Hands: A Sensory Substitution That Supports Manual InteractionsSensory-substitution devices enable perceiving objects by translating one modality (e.g., vision) into another (e.g., tactile). While many explored the placement of the haptic-output (e.g., torso, forehead), the camera’s location remains largely unexplored—typically seeing from the eyes’ perspective. Instead, we propose that seeing & feeling information from the hands’ perspective could enhance flexibility & expressivity of sensory-substitution devices to support manual interactions with physical objects. To this end, we engineered a back-of-the-hand electrotactile-display that renders tactile images from a wrist-mounted camera, allowing the user’s hand to feel objects while reaching & hovering. We conducted a study with sighted/Blind-or-Low-Vision participants who used our eyes vs. hand tactile-perspectives to manipulate bottles and soldering-irons, etc. We found that while both tactile perspectives provided comparable performance, when offered the opportunity to choose, all participants found value in also using the hands’ perspective. Moreover, we observed behaviors when “seeing with the hands” that suggest a more ergonomic object-manipulation. We believe these insights extend the landscape of sensory-substitution devices.2025STShan-Yuan Teng et al.University of ChicagoIn-Vehicle Haptic, Audio & Multimodal FeedbackVibrotactile Feedback & Skin StimulationVisual Impairment Technologies (Screen Readers, Tactile Graphics, Braille)CHI
Can a Smartwatch Move Your Fingers? Compact and Practical Electrical Muscle Stimulation in a SmartwatchSmartwatches gained popularity in the mainstream, making them into today’s de-facto wearables. Despite advancements in sensing, haptics on smartwatches is still restricted to tactile feedback (e.g., vibration). Most smartwatch-sized actuators cannot render strong force-feedback. Simultaneously, electrical muscle stimulation (EMS) promises compact force-feedback but, to actuate fingers requires users to wear many electrodes on their forearms. While forearm electrodes provide good accuracy, they detract EMS from being a practical force-feedback interface. To address this, we propose moving the electrodes to the wrist—conveniently packing them in the backside of a smartwatch. In our first study, we found that by cross-sectionally stimulating the wrist in 1,728 trials, we can actuate thumb extension, index extension & flexion, middle flexion, pinky flexion, and wrist flexion. Following, we engineered a compact EMS that integrates directly into a smartwatch’s wristband (with a custom stimulator, electrodes, demultiplexers, and communication). In our second study, we found that participants could calibrate our device by themselves ~50% faster than with conventional EMS. Furthermore, all participants preferred the experience of this device, especially for its social acceptability & practicality. We believe that our approach opens new applications for smartwatch-based interactions, such as haptic assistance during everyday tasks.2024ATAkifumi Takahashi et al.Electrical Muscle Stimulation (EMS)Smartwatches & Fitness BandsUIST
Haptic Permeability: Adding Holes to Tactile Devices Improves DexterityFeeling haptics with our fingerpads is how we achieve manual tasks (e.g., operate a needle or press buttons). Following this, research started adding actuators atop the users’ fingerpads to render haptic feedback for interactive virtual environments. Recently, many have moved away from thick actuators (e.g., vibration motors) and turned to electrode-films with electrotactile stimulation—allowing users to still feel some sensations through the devices when touching physical objects (e.g., compliance or some macro features). However, we argue & demonstrate that thin devices are not enough to maximize the user’s dexterity. We evaluate how adding small holes to electrotactile films can allow direct contact and thus increase haptic permeability, resulting in: (1) improved perception of tactile features; and (2) improved force control in grasping tasks. Finally, we observed participants in interactive experiences and found that holes can preserve dexterity with physical tasks while still benefiting from haptic feedback.2024STShan-Yuan Teng et al.University of ChicagoVibrotactile Feedback & Skin StimulationForce Feedback & Pseudo-Haptic WeightCHI
ThermalRouter: Enabling Users to Design Thermally-Sound Devices Users often 3D model enclosures that interact with significant heat sources, such as electronics or appliances that generate heat (e.g., CPU, motor, lamps, etc.). While parts made by users might function well aesthetically or structurally, they are rarely thermally-sound. This happens because heat transfer is non-intuitive; thus, engineering thermal solutions is not straightforward. To tackle this, we developed ThermalRouter, a CAD plugin that assists with improving the thermal performance of their models. ThermalRouter automatically converts regions of the model to be made from thermally-conductive materials (such as nylon or metallic-silicone). These regions act as heat channels, branching away from hotspots to dissipate heat. The key is that ThermalRouter automatically simulates the thermal performance of many possible heat channel configurations and presents the user with the most thermally-sound design (e.g., lowest temperature). Furthermore, it allows users to customize by balancing costs, indicating non-modifiable geometry, etc. Most importantly, ThermalRouter achieves this without requiring manual labor to set up or parse the results of complex thermal simulations.2023AMAlex Mazursky et al.Laser Cutting & Digital FabricationCircuit Making & Hardware PrototypingCustomizable & Personalized ObjectsUIST
Prolonging VR Haptic Experiences by Harvesting Kinetic Energy from the UserWe propose a new technical approach to implement untethered VR haptic devices that contain no battery, yet can render on-demand haptic feedback. The key is that via our approach, a haptic device charges itself by harvesting the user’s kinetic energy (i.e., movement)—even without the user needing to realize this. This is achieved by integrating the energy-harvesting with the virtual experience, in a responsive manner. Whenever our batteryless haptic device is about to lose power, it switches to harvesting mode (by engaging its clutch to a generator) and, simultaneously, the VR headset renders an alternative version of the current experience that depicts resistive forces (e.g., rowing a boat in VR). As a result, the user feels realistic haptics that corresponds to what they should be feeling in VR, while unknowingly charging the device via their movements. Once the haptic device’s supercapacitors are charged, they wake up its microcontroller to communicate with the VR headset. The VR experience can now use the recently harvested power for on-demand haptics, including vibration, electrical or mechanical force-feedback; this process can be repeated, ad infinitum. We instantiated a version of our concept by implementing an exoskeleton (with vibration, electrical & mechanical force-feedback) that harvests the user’s arm movements. We validated it via a user study, in which participants, even without knowing the device was harvesting, rated its’ VR experience as more realistic & engaging than with a baseline VR setup. Finally, we believe our approach enables haptics for prolonged uses, especially useful in untethered VR setups, since devices capable of haptic feedback are traditionally only reserved for situations with ample power. Instead, with our approach, a user who engages in hours-long VR and grew accustomed to finding a battery-dead haptic device that no longer works, will simply resurrect the haptic device with their movement.2022STShan-Yuan Teng et al.Haptic WearablesShape-Changing Interfaces & Soft Robotic MaterialsFull-Body Interaction & Embodied InputUIST
Altering Perceived Softness of Real Rigid Objects by Restricting Fingerpad DeformationWe propose a haptic device that alters the perceived softness of real rigid objects without requiring to instrument the objects. Instead, our haptic device works by restricting the user’s fingerpad lateral deformation via a hollow frame that squeezes the sides of the fingerpad. This causes the fingerpad to become bulgier than it originally was—when users touch an object’s surface with their now-restricted fingerpad, they feel the object to be softer than it is. To illustrate the extent of softness illusion induced by our device, touching the tip of a wooden chopstick will feel as soft as a rubber eraser. Our haptic device operates by pulling the hollow frame using a motor. Unlike most wearable haptic devices, which cover up the user’s fingerpad to create force sensations, our device creates softness while leaving the center of the fingerpad free, which allows the users to feel most of the object they are interacting with. This makes our device a unique contribution to altering the softness of everyday objects, creating “buttons” by softening protrusions of existing appliances or tangibles, or even, altering the softness of handheld props for VR. Finally, we validated our device through two studies: (1) a psychophysics study showed that the device brings down the perceived softness of any object between 50A-90A to around 40A (on Shore A hardness scale); and (2) a user study demonstrated that participants preferred our device for interactive applications that leverage haptic props, such as making a VR prop feel softer or making a rigid 3D printed remote control feel softer on its button.2021YTYujie Tao et al.In-Vehicle Haptic, Audio & Multimodal FeedbackVibrotactile Feedback & Skin StimulationForce Feedback & Pseudo-Haptic WeightUIST
DextrEMS: Achieving Dexterity in Electrical Muscle Stimulation by Combining it with BrakesElectrical muscle stimulation (EMS) is an emergent technique that miniaturizes force feedback, especially popular for untethered haptic devices, such as mobile gaming, VR, or AR. However, the actuation displayed by interactive systems based on EMS is coarse and imprecise. EMS systems mostly focus on inducing movements in large muscle groups such as legs, arms, and wrists; whereas individual finger poses, which would be required, for example, to actuate a user’s fingers to fingerspell even the simplest letters in sign language, are not possible. The lack of dexterity in EMS stems from two fundamental limitations: (1) lack of independence: when a particular finger is actuated by EMS, the current runs through nearby muscles, causing unwanted actuation of adjacent fingers; and, (2) unwanted oscillations: while it is relatively easy for EMS to start moving a finger, it is very hard for EMS to stop and hold that finger at a precise angle; because, to stop a finger, virtually all EMS systems contract the opposing muscle, typically achieved via controllers (e.g., PID)—unfortunately, even with the best controller tuning, this often results in unwanted oscillations. To tackle these limitations, we propose dextrEMS, an EMS-based haptic device featuring mechanical brakes attached to each finger joint. The key idea behind dextrEMS is that while the EMS actuates the fingers, it is our mechanical brake that stops the finger in a precise position. Moreover, it is also the brakes that allow dextrEMS to select which fingers are moved by EMS, eliminating unwanted movements by preventing adjacent fingers from moving. We implemented dextrEMS as an untethered haptic device, weighing only 68g, that actuates eight finger joints independently (metacarpophalangeal and proximal interphalangeal joints for four fingers), which we demonstrate in a wide range of haptic applications, such as assisted fingerspelling, a piano tutorial, guitar tutorial, and a VR game. Finally, in our technical evaluation, we found that dextrEMS outperformed EMS alone by doubling its independence and reducing unwanted oscillations.2021RNRomain Nith et al.Electrical Muscle Stimulation (EMS)Haptic WearablesUIST
MagnetIO: Passive yet Interactive Soft Haptic Patches Anywhere We propose a new type of haptic actuator, which we call MagnetIO, that is comprised of two parts: one battery-powered voice-coil worn on the user’s fingernail and any number of interactive soft patches that can be attached onto any surface (everyday objects, user’s body, appliances, etc.). When the user’s finger wearing our voice-coil contacts any of the interactive patches it detects its magnetic signature via magnetometer and vibrates the patch, adding haptic feedback to otherwise input-only interactions. To allow these passive patches to vibrate, we make them from silicone with regions doped with polarized neodymium powder, resulting in soft and stretchable magnets. This stretchable form-factor allows them to be wrapped to the user’s body or everyday objects of various shapes. We demonstrate how these add haptic output to many situations, such as adding haptic buttons to the walls of one’s home. In our technical evaluation, we demonstrate that our interactive patches can be excited across a wide range of frequencies (0-500 Hz) and can be tuned to resonate at specific frequencies based on the patch’s geometry. Furthermore, we demonstrate that MagnetIO’s vibration intensity is as powerful as a typical linear resonant actuator (LRA); yet, unlike these rigid actuators, our passive patches operate as springs with multiple modes of vibration, which enables a wider band around its resonant frequency than an equivalent LRA.2021AMAlex Mazursky et al.University of ChicagoVibrotactile Feedback & Skin StimulationHaptic WearablesShape-Changing Interfaces & Soft Robotic MaterialsCHI
Elevate: A Walkable Pin-Array for Large Shape-Changing TerrainsCurrent head-mounted displays enable users to explore virtual worlds by simply walking through them (i.e., real-walking VR). This led researchers to create haptic displays that can also simulate different types of elevation shapes. However, existing shape-changing floors are limited by their tabletop scale or the coarse resolution of the terrains they can display due to the limited number of actuators and low vertical resolution. To tackle this challenge, we introduce Elevate, a dynamic and walkable pin-array floor on which users can experience not only large variations in shapes but also the details of the underlying terrain. Our system achieves this by packing 1200 pins arranged on a 1.80 x 0.60m platform, in which each pin can be actuated to one of ten height levels (resolution: 15mm/level). To demonstrate its applicability, we present our haptic floor combined with four walkable applications and a user study that reported increased realism and enjoyment.2021SJSeungwoo Je et al.KAISTShape-Changing Interfaces & Soft Robotic MaterialsFull-Body Interaction & Embodied InputCHI
Stereo-Smell via Electrical Trigeminal StimulationWe propose a novel type of olfactory device that creates a stereo-smell experience, i.e., directional information about the location of an odor, by rendering the readings of external odor sensors as trigeminal sensations using electrical stimulation of the user’s nasal septum. The key is that the sensations from the trigeminal nerve, which arise from nerve-endings in the nose, are perceptually fused with those of the olfactory bulb (the brain region that senses smells). As such, we propose that electrically stimulating the trigeminal nerve is an ideal candidate for stereo-smell augmentation/substitution that, unlike other approaches, does not require implanted electrodes in the olfactory bulb. To realize this, we engineered a self-contained device that users wear across their nasal septum. Our device outputs by stimulating the user’s trigeminal nerve using electrical impulses with variable pulse-widths; and it inputs by sensing the user’s inhalations using a photoreflector. It measures 10x23 mm and communicates with external gas sensors using Bluetooth. In our user study, we found the key electrical waveform parameters that enable users to feel an odor’s intensity (absolute electric charge) and direction (phase order and net charge). In our second study, we demonstrated that participants were able to localize a virtual smell source in the room by using our prototype without any previous training. Using these insights, our device enables expressive trigeminal sensations and could function as an assistive device for people with anosmia, who are unable to smell.2021JBJas Brooks et al.University of ChicagoElectrical Muscle Stimulation (EMS)Biosensors & Physiological MonitoringCHI
Touch&Fold: A Foldable Haptic Actuator for Rendering Touch in Mixed RealityWe propose a nail-mounted foldable haptic device that provides tactile feedback to mixed reality (MR) environments by pressing against the user’s fingerpad when a user touches a virtual object. What is novel in our device is that it quickly tucks away when the user interacts with real-world objects. Its design allows it to fold back on top of the user’s nail when not in use, keeping the user’s fingerpad free to, for instance, manipulate handheld tools and other objects while in MR. To achieve this, we engineered a wireless and self-contained haptic device, which measures 24×24×41 mm and weighs 9.5 g. Furthermore, our foldable end-effector also features a linear resonant actuator, allowing it to render not only touch contacts (i.e., pressure) but also textures (i.e., vibrations). We demonstrate how our device renders contacts with MR surfaces, buttons, low- and high-frequency textures. In our first user study, we found that participants perceived our device to be more realistic than a previous haptic device that also leaves the fingerpad free (i.e., fingernail vibration). In our second user study, we investigated the participants’ experience while using our device in a real-world task that involved physical objects. We found that our device allowed participants to use the same finger to manipulate handheld tools, small objects, and even feel textures and liquids, without much hindrance to their dexterity, while feeling haptic feedback when touching MR interfaces.2021STShan-Yuan Teng et al.University of ChicagoHaptic WearablesShape-Changing Interfaces & Soft Robotic MaterialsMixed Reality WorkspacesCHI
HandMorph: a Passive Exoskeleton that Miniaturizes GraspWe engineered an exoskeleton, which we call HandMorph, that approximates the experience of having a smaller grasping range. It uses mechanical links to transmit motion from the wearer’s fingers to a smaller hand with five anatomically correct fingers. The result is that HandMorph miniaturizes a wearer’s grasping range while transmitting haptic feedback. Unlike other size-illusions based on virtual reality, HandMorph achieves this in the user’s real environment, preserving the user’s physical and social contexts. As such, our device can be integrated into the user’s workflow, e.g., to allow product designers to momentarily change their grasping range into that of a child while evaluating a toy prototype. In our first user study, we found that participants perceived objects as larger when wearing HandMorph, which suggests that their size perception was successfully transformed. In our second user study, we assessed the experience of using HandMorph in designing a simple toy trumpet for children. We found that participants felt more confident in their toy design when using HandMorph to validate its ergonomics.2020JNJun Nishida et al.Shape-Changing Interfaces & Soft Robotic MaterialsHand Gesture RecognitionUIST
Wearable Microphone JammingWe engineered a wearable microphone jammer that is capable of disabling microphones in its user's surroundings, including hidden microphones. Our device is based on a recent exploit that leverages the fact that when exposed to ultrasonic noise, commodity microphones will leak the noise into the audible range.<br>Unfortunately, ultrasonic jammers are built from multiple transducers and therefore exhibit blind spots, i.e., locations in which transducers destructively interfere and where a microphone cannot be jammed. To solve this, our device exploits a synergy between ultrasonic jamming and the naturally occur- ring movements that users induce on their wearable devices (e.g., bracelets) as they gesture or walk. We demonstrate that these movements can blur jamming blind spots and increase jamming coverage. Moreover, current jammers are also directional, requiring users to point the jammer to a microphone; instead, our wearable bracelet is built in a ring-layout that al- lows it to jam in multiple directions. This is beneficial in that it allows our jammer to protect against microphones hidden out of sight.<br>We evaluated our jammer in a series of experiments and found that: (1) it jams in all directions, e.g., our device jams over 87% of the words uttered around it in any direction, while existing devices jam only 30% when not pointed directly at the microphone; (2) it exhibits significantly less blind spots; and, (3) our device induced a feeling of privacy to participants of our user study. We believe our wearable provides stronger privacy in a world in which most devices are constantly eavesdropping on our conversations.2020YCYuxin Chen et al.University of ChicagoPrivacy by Design & User ControlIoT Device PrivacySmart Home Privacy & SecurityCHI