ReflecTrace: Touchless Hover Interaction on Commodity Smartphones via Corneal ReflectionWe propose an approach to detect finger hover inputs on a smartphone screen using corneal reflection images captured by the device’s built-in front camera. This method requires no external sensors or hardware, enabling hover input detection in the near-screen space that is not directly visible to the camera. By leveraging a convolutional neural network (CNN), we estimate the two-dimensional position of a hovering finger and classify it into a predefined screen grid. Experimental results show that our model achieves approximately 95% accuracy for coarse grids and maintains over 88% accuracy for finer divisions. Furthermore, our system demonstrates real-time processing capability with an end-to-end latency of approximately 22 ms on a standard smartphone. These findings highlight the practical feasibility of camera-only hover sensing and suggest a wide range of touchless interaction applications, enabling touchless interaction when touch is undesirable, pre-touch UI adaptation, and accessibility support on commodity mobile devices.2026YNYudai Nakamura et al.Keio UniversityMulti-Touch Interaction TechniquesMobile App User ExperienceMobile Accessibility DesignIUI
Zip-up Print: Rapid and Assemblable 3D printing Using 2D Flattened Zipper-like StructuresWe propose a method to fabricate objects composed of 3D printed flattened pieces with integrated zipper-like structures. The object is manually assembled into a 3D shape by connecting the zipper components. By employing a zipper design that allows for angle-independent connections between patches, our method enables both the surface and zipper components to be printed in the same orientation, resulting in high-quality reconstruction of the input model with a faster 3D printing process that wastes less material. We implement a fully automated pipeline that takes a 3D model as input, converts it into developable patches, generates the zipper structures, and flattens them for subsequent 3D printing. We demonstrate that our approach significantly reduces the fabrication time and support material consumption. We also present application examples that highlight the versatility of our method.2026TYTakumi Yamamoto et al.Keio UniversityDesktop 3D Printing & Personal FabricationShape-Changing Materials & 4D PrintingCHI
SoilSense: Appropriating Soil-based Microbial Fuel Cells to Create Tangible InterfacesSoil-based Microbial Fuel Cells (SMFCs) offer a sustainable method for powering low-energy computing devices by harnessing electricity from microbial activity in soil. In this paper, we introduce SoilSense, a novel approach that repurposes SMFCs as tangible interfaces, transforming soil into an interactive, computationally responsive medium, instead of energy sources. We explore the voltage variations that occur when pressure is applied to the cathode and systematically characterize this mechanism across different electrode configurations and soil moisture levels. To demonstrate the feasibility of SMFC-based interfaces, we present a series of modular and proof-of-concept prototypes that support diverse interaction modalities. We further illustrate how SoilSense enables interactions through example applications and provide implications and envision for future studies to employ soil as an ecologically compatible material in interactive system design.2025TMTian Min et al.Shape-Changing Materials & 4D PrintingEcological Design & Green ComputingEnergy Conservation Behavior & InterfacesUIST
ScanRing: Hybrid Authentication System in a Ring Device Using a Distance Sensor and an IMU SensorSmart rings are used for contactless payment, smart lock operation, and health monitoring. For applications such as electronic payment and unlocking smart locks, the implementation of a user authentication system in smart rings is essential; however, some challenges remain. Fingerprint authentication is sensitive to fingertip conditions, while face authentication faces difficulties with miniaturization, power efficiency, and privacy. This study proposes ScanRing, a hybrid authentication system using a distance sensor and an IMU sensor in a smart ring. By moving the ring device laterally in front of the face, the distance sensor captures facial structure data, while the IMU sensor records the user’s motion characteristics. These combined datasets enable robust user authentication without relying on cameras, which enhances privacy while supporting a compact and power-efficient design. A user study (N = 30) demonstrated that ScanRing achieved an average authentication accuracy of 98.41 % under stable conditions.2025KMKai Miyashita et al.Passwords & AuthenticationMobileHCI
MaGEL: A Soft, Transparent Input Device Enabling Deformation Gesture RecognitionWe propose MaGEL, a soft-input device that utilizes light intensity to detect and interpret user deformation interactions. Unlike traditional rigid input devices, MaGEL enables three-dimensional interactions such as twisting, bending, and pulling. Additionally, MaGEL incorporates elastic haptic feedback, providing users with tactile sensations that reflect the tension or resistance of their interactions. These factors realize intuitive and natural user interaction experiences, and users can employ familiar physical gestures as input. For example, bending the device may simulate turning the page of a book, or stretching it may zoom in on an image. The device consists of a transparent urethane resin gel with LED lights and phototransistors on both sides. When the device gel deforms, the intensity of the light passing through the gel undergoes a specific change due to the deformation. The system analyzes these changes using machine learning to identify the user gestures. We evaluated the optimal configuration and number of LEDs and phototransistors to classify the deformation accurately. We acquired data for 13 types of deformation gestures from 14 participants. The results showed that a combination of four LEDs and ten phototransistors enabled MaGEL to identify 13 types of deformation gestures with an accuracy of 94.1 %. Using MaGEL, we provide novel interactive experiences, such as game controllers that employ bending, pulling, or twisting to mimic natural gaming motions.2025FOFumika Oguri et al.Shape-Changing Interfaces & Soft Robotic MaterialsHand Gesture RecognitionIUI
FlexEar-Tips: Shape-Adjustable Ear Tips Using Pressure ControlWe introduce FlexEar-Tips, a dynamic ear tip system designed for the next-generation hearables. The ear tips are controlled by an air pump and solenoid valves, enabling size adjustments for comfort and functionality. FlexEar-Tips includes an air pressure sensor to monitor ear tip size, allowing it to adapt to environmental conditions and user needs. In the evaluation, we conducted a preliminary investigation of the size control accuracy and the minimum amount of variability of haptic perception in the user's ear. We then evaluated the user's ability to identify patterns in the haptic notification system, the impact on the music listening experience, the relationship between the size of the ear tips and the sound localization ability, and the impact on the reduction of humidity in the ear using a model. We proposed new interaction modalities for adaptive hearables and discussed health monitoring, immersive auditory experiences, haptics notifications, biofeedback, and sensing.2025TATakashi Amesaka et al.Keio University, Lifestyle Computing LabHaptic WearablesShape-Changing Interfaces & Soft Robotic MaterialsCHI
IrOnTex: Using Ironable 3D Printed Objects to Fabricate and Prototype Customizable Interactive TextilesYu等人提出IrOnTex，利用可熨烫3D打印对象制作定制交互式纺织品，实现快速原型设计。2024JYJiakun Yu et al.Desktop 3D Printing & Personal FabricationTextile Art & Craft DigitizationUbiComp
EarHover: Mid-Air Gesture Recognition for Hearables Using Sound Leakage SignalsWe introduce EarHover, an innovative system that enables mid-air gesture input for hearables. Mid-air gesture input, which eliminates the need to touch the device and thus helps to keep hands and the device clean, has been known to have high demand based on previous surveys. However, existing mid-air gesture input methods for hearables have been limited to adding cameras or infrared sensors. By focusing on the sound leakage phenomenon unique to hearables, we have realized mid-air gesture recognition using a speaker and an external microphone that are highly compatible with hearables. The signal leaked to the outside of the device due to sound leakage can be measured by an external microphone, which detects the differences in reflection characteristics caused by the hand's speed and shape during mid-air gestures. Among 27 types of gestures, we determined the seven most suitable gestures for EarHover in terms of signal discrimination and user acceptability. We then evaluated the gesture detection and classification performance of two prototype devices (in-ear type/open-ear type) for real-world application scenarios.2024SSShunta Suzuki et al.In-Vehicle Haptic, Audio & Multimodal FeedbackHand Gesture RecognitionUIST
Exploring User-Defined Gestures as Input for Hearables and Recognizing of Ear-Touch Gestures by IMUsHearables are highly functional earphone-type wearables; however, existing input methods using stand-alone hearables are limited in the number of commands, and there is a need to extend device operation through hand gestures. In previous research on hearables for hand input, user understanding and gesture recognition systems have been developed. However, in the realm of user understanding, exploration remains incomplete concerning hand input with hearables, and extant recognition systems have not demonstrated proficiency in discerning user-defined gestures. In this study, we conducted a gesture elicitation study (GES) assuming hand input using hearables under six conditions (three interaction areas × two device shapes). Then, we extracted ear-touch gestures that the device's built-in IMU sensor could recognize from the user-defined gestures and investigated recognition performance. The results of the experiments in a sitting experiment showed that the gesture recognition rate for in-ear devices was 91.0%, and for ear-hook devices was 74.7%.2024YSYukina Sato et al.Vibrotactile Feedback & Skin StimulationHand Gesture RecognitionMobileHCI
AudioMove: Applying the Spatial Audio to Multi-Directional Limb Exercise GuidanceGuiding users with limb exercise can assist muscle training or physical recovery. However, traditional vision-based methods often require multiple camera angles to help users understand the motions and require them to be within the range of the screen. Therefore, we propose a non-visual system that can guide users with multiple-directional limb motions utilizing spatial audio, AudioMove, with the commercial-off-the-shelf (COTS) devices (i.e., smartphones and earphones). The proposed system addresses the challenge of conveying directional information encompassing multiple planes in real time. We conduct a mix-method user study to evaluate the effectiveness of the system with three methods combining motion data with spatial audio perception. Additionally, a user interface is built for collecting users' comments. The results conclude that spatial audio guidance could create a natural, pervasive, and non-visual exercise training solution in daily life.2024CXChengshuo Xia et al.In-Vehicle Haptic, Audio & Multimodal FeedbackFull-Body Interaction & Embodied InputMobileHCI
TouchLog: Finger Micro Gesture Recognition Using Photo-Reflective SensorsFingertip input allows for interactions that are natural, easy to perform, and socially acceptable. It also has advantages in terms of low physical demand, confidentiality, and haptic feedback. In this study, we propose TouchLog, a fingernail-type device that uses skin deformation of the fingertip to identify finger micro gestures written with the thumb on the index finger. TouchLog is attached to the index fingernail and allows for one-handed fingertip input without compromising the haptic feedback on the finger. To evaluate the accuracy of 11 types of finger micro gesture recognition, we conducted a user study (N = 10) and obtained an average identification accuracy of 91.5\% (SD = 3.1\%). A continuous input method using skin deformation and contact pressure was also examined, and its usefulness as a wearable device was discussed.2023RKRiku Kitamura et al.Vibrotactile Feedback & Skin StimulationHaptic WearablesHand Gesture RecognitionUbiComp
User Authentication Method for Hearables Using Sound Leakage SignalsWe propose a novel biometric authentication method that leverages sound leakage signals from hearables that are captured by an external microphone. A sweep signal is played from hearables, and sound leakage is recorded using an external microphone. This sound leakage signal represents the acoustic characteristics of the ear canal, auricle, or hand. Then, our system analyzes the echoes and authenticates the user. The proposed method is highly adaptable to hearables because it leverages widely available sensors, such as speakers and external microphones. In addition, the proposed method has the potential to be used in combination with existing methods. In this study, we investigate the characteristics of sound leakage signals using an experimental model and measure the authentication performance of our method using acoustic data from 16 people. The results show that the balanced accuracy (BAC) scores were in the range of 87.0%-96.7% in several scenarios.2023TATakashi Amesaka et al.Passwords & AuthenticationUbiComp
ReflecTouch: Detecting Grasp Posture of Smartphone Using Corneal Reflection ImagesBy sensing how a user is holding a smartphone, adaptive user interfaces are possible such as those that automatically switch the displayed content and position of graphical user interface (GUI) components following how the phone is being held. We propose ReflecTouch, a novel method for detecting how a smartphone is being held by capturing images of the smartphone screen reflected on the cornea with a built-in front camera. In these images, the areas where the user places their fingers on the screen appear as shadows, which makes it possible to estimate the grasp posture. Since most smartphones have a front camera, this method can be used regardless of the device model; in addition, no additional sensor or hardware is required. We conducted data collection experiments to verify the classification accuracy of the proposed method for six different grasp postures, and the accuracy was 85%.2022XZXiang Zhang et al.Keio UniversityHuman Pose & Activity RecognitionPrototyping & User TestingCHI