NeuResonance: Exploring Feedback Experiences for Fostering the Inter-brain SynchronizationWhen several individuals collaborate on a shared task, their brain activities often synchronize. This phenomenon, known as Inter-brain Synchronization (IBS), is notable for inducing prosocial outcomes such as enhanced interpersonal feelings, including closeness, trust, empathy, and more. Further strengthening the IBS with the aid of external feedback would be beneficial for scenarios where those prosocial feelings play a vital role in interpersonal communication, such as rehabilitation between a therapist and a patient, motor skill learning between a teacher and a student, and group performance art. This paper investigates whether visual, auditory, and haptic feedback of the IBS level can further enhance its intensity, offering design recommendations for feedback systems in IBS. We report findings when three different types of feedback were provided: IBS level feedback by means of on-body projection mapping, sonification using chords, and vibration bands attached to the wrist.2025JDJamie Ngoc Dinh et al.University of Maryland, College Park, College of Information StudiesMid-Air Haptics (Ultrasonic)Vibrotactile Feedback & Skin StimulationBrain-Computer Interface (BCI) & NeurofeedbackCHI
DigituSync: A Dual-User Passive Exoskeleton Glove That Adaptively Shares Hand GesturesWe engineered DigituSync, a passive-exoskeleton that physically links two hands together, enabling two users to adaptively transmit finger movements in real-time. It uses multiple four-bar linkages to transfer both motion and force, while still preserving congruent haptic feedback. Moreover, we implemented a variable-length linkage that allows adjusting the force transmission ratio between the two users and regulates the amount of intervention, which enables users to customize their learning experience. DigituSync's benefits emerge from its passive design: unlike existing haptic devices (motor-based exoskeletons or electrical muscle stimulation), DigituSync has virtually no latency and does not require batteries/electronics to transmit or adjust movements, making it useful and safe to deploy in many settings, such as between students and teachers in a classroom. We validated DigituSync by means of technical evaluations and a user study, demonstrating that it instantly transfers finger motions and forces with the ability of adaptive force transmission, which allowed participants to feel more control over their own movements and to feel the teacher’s intervention was more responsive. We also conducted two exploratory sessions with a music teacher and deaf-blind users, which allowed us to gather experiential insights from the teacher’s side and explore DigituSync in applications.2022JNJun Nishida et al.Force Feedback & Pseudo-Haptic WeightHaptic WearablesUIST
Electrical Head Actuation: Enabling Interactive Systems to Directly Manipulate Head OrientationWe propose a novel interface concept in which interactive systems directly manipulate the user’s head orientation. We implement this using electrical-muscle-stimulation (EMS) of the neck muscles, which turns the head around its yaw (left/right) and pitch (up/down) axis. As the first exploration of EMS for head actuation, we characterized which muscles can be robustly actuated. Second, we evaluated the accuracy of our system for actuating participants' head orientation towards static targets and trajectories. Third, we demonstrated how it enables interactions not possible before by building a range of applications, such as (1) synchronizing head orientations of two users, which enables a user to communicate head nods to another user while listening to music, and (2) directly changing the user's head orientation to locate objects in AR. Finally, in our second study, participants felt that our head actuation contributed positively to their experience in four distinct applications.2022YTYudai Tanaka et al.University of ChicagoElectrical Muscle Stimulation (EMS)Human Pose & Activity RecognitionBrain-Computer Interface (BCI) & NeurofeedbackCHI
Stereo-Smell via Electrical Trigeminal StimulationWe propose a novel type of olfactory device that creates a stereo-smell experience, i.e., directional information about the location of an odor, by rendering the readings of external odor sensors as trigeminal sensations using electrical stimulation of the user’s nasal septum. The key is that the sensations from the trigeminal nerve, which arise from nerve-endings in the nose, are perceptually fused with those of the olfactory bulb (the brain region that senses smells). As such, we propose that electrically stimulating the trigeminal nerve is an ideal candidate for stereo-smell augmentation/substitution that, unlike other approaches, does not require implanted electrodes in the olfactory bulb. To realize this, we engineered a self-contained device that users wear across their nasal septum. Our device outputs by stimulating the user’s trigeminal nerve using electrical impulses with variable pulse-widths; and it inputs by sensing the user’s inhalations using a photoreflector. It measures 10x23 mm and communicates with external gas sensors using Bluetooth. In our user study, we found the key electrical waveform parameters that enable users to feel an odor’s intensity (absolute electric charge) and direction (phase order and net charge). In our second study, we demonstrated that participants were able to localize a virtual smell source in the room by using our prototype without any previous training. Using these insights, our device enables expressive trigeminal sensations and could function as an assistive device for people with anosmia, who are unable to smell.2021JBJas Brooks et al.University of ChicagoElectrical Muscle Stimulation (EMS)Biosensors & Physiological MonitoringCHI
Preserving Agency During Electrical Muscle Stimulation Training Speeds up Reaction Time Directly After Removing EMSForce feedback devices, such as motor-based exoskeletons or wearables based on electrical muscle stimulation (EMS), have the unique potential to accelerate users’ own reaction time (RT). However, this speedup has only been explored while the device is attached to the user. In fact, very little is known regarding whether this faster reaction time still occurs after the user removes the device from their bodies–this is precisely what we investigated by means of a simple reaction time (RT) experiment, in which participants were asked to tap as soon as they saw an LED flashing. Participants experienced this in three EMS conditions: (1) fast-EMS, the electrical impulses were synced with the LED; (2) agency-EMS, the electrical impulse was delivered 40ms faster than the participant’s own RT, which prior work has shown to preserve one’s sense of agency over this movement; and, (3) late-EMS: the impulse was delivered after the participant’s own RT. Our results revealed that the participants’ RT was significantly reduced by approximately 8ms(up to 20ms) only after training with the agency-EMS condition. This finding suggests that the prioritizing agency during EMS training is key to motor-adaptation, i.e., it enables a faster motor response even after the user has removed the EMS device from their body.2021SKShunichi Kasahara et al.Sony CSL, The University of TokyoVibrotactile Feedback & Skin StimulationElectrical Muscle Stimulation (EMS)CHI
HandMorph: a Passive Exoskeleton that Miniaturizes GraspWe engineered an exoskeleton, which we call HandMorph, that approximates the experience of having a smaller grasping range. It uses mechanical links to transmit motion from the wearer’s fingers to a smaller hand with five anatomically correct fingers. The result is that HandMorph miniaturizes a wearer’s grasping range while transmitting haptic feedback. Unlike other size-illusions based on virtual reality, HandMorph achieves this in the user’s real environment, preserving the user’s physical and social contexts. As such, our device can be integrated into the user’s workflow, e.g., to allow product designers to momentarily change their grasping range into that of a child while evaluating a toy prototype. In our first user study, we found that participants perceived objects as larger when wearing HandMorph, which suggests that their size perception was successfully transformed. In our second user study, we assessed the experience of using HandMorph in designing a simple toy trumpet for children. We found that participants felt more confident in their toy design when using HandMorph to validate its ergonomics.2020JNJun Nishida et al.Shape-Changing Interfaces & Soft Robotic MaterialsHand Gesture RecognitionUIST
Next Steps for Human-Computer IntegrationHuman-Computer Integration (HInt) is an emerging paradigm in which computational and human systems are closely interwoven. Integrating computers with the human body is not new. however, we believe that with rapid technological advancements, increasing real-world deployments, and growing ethical and societal implications, it is critical to identify an agenda for future research. We present a set of challenges for HInt research, formulated over the course of a five-day workshop consisting of 29 experts who have designed, deployed and studied HInt systems. This agenda aims to guide researchers in a structured way towards a more coordinated and conscientious future of human-computer integration.2020FMFlorian Floyd Mueller et al.Monash UniversityBrain-Computer Interface (BCI) & NeurofeedbackTechnology Ethics & Critical HCIUser Research Methods (Interviews, Surveys, Observation)CHI
Preemptive Action: Accelerating Human Reaction using Electrical Muscle Stimulation Without Compromising AgencyWe enable preemptive force-feedback systems to speed up human reaction time without fully compromising the user's sense of agency. Typically these interfaces actuate by means of electrical muscle stimulation (EMS) or mechanical actuators; they preemptively move the user to perform a task, such as to improve movement performance (e.g., EMS-assisted drumming). Unfortunately, when using preemptive force-feedback users do not feel in control and loose their sense of agency. We address this by actuating the user's body, using EMS, within a particular time window (160 ms after visual stimulus), which we found to speed up reaction time by 80 ms in our first study. With this preemptive timing, when the user and system move congruently, the user feels that they initiated the motion, yet their reaction time is faster than usual. As our second study demonstrated, this particular timing significantly increased agency when compared to the current practice in EMS-based devices. We conclude by illustrating, using examples from the HCI literature, how to leverage our findings to provide more agency to automated haptic interfaces.2019SKShunichi Kasahara et al.Sony CSL & University of TokyoForce Feedback & Pseudo-Haptic WeightElectrical Muscle Stimulation (EMS)CHI
Egocentric Smaller-person Experience through a Change in Visual PerspectiveThis paper explores how human perceptions, actions, and interactions can be changed through an embodied and active experience of being a smaller person in a real-world environment, which we call an egocentric smaller person experience. We developed a wearable visual translator that provides the perspective of a smaller person by shifting the wearer's eyesight level down to their waist using a head-mounted display and a stereo camera module, while allowing for field of view control through head movements. In this study, we investigated how the developed device can modify the wearer's body representation and experiences based on a field study conducted at a nursing school and museums, and through lab studies. It was observed that the participants changed their perceptions, actions, and interactions because they are considered to have perceived themselves as being smaller. Using this device, designers and teachers can understand the perspectives of other people in an existing environment.2019JNJun Nishida et al.JSPS & University of ChicagoFull-Body Interaction & Embodied InputEye Tracking & Gaze InteractionIdentity & Avatars in XRCHI