ReflecTrace: Touchless Hover Interaction on Commodity Smartphones via Corneal ReflectionWe propose an approach to detect finger hover inputs on a smartphone screen using corneal reflection images captured by the device’s built-in front camera. This method requires no external sensors or hardware, enabling hover input detection in the near-screen space that is not directly visible to the camera. By leveraging a convolutional neural network (CNN), we estimate the two-dimensional position of a hovering finger and classify it into a predefined screen grid. Experimental results show that our model achieves approximately 95% accuracy for coarse grids and maintains over 88% accuracy for finer divisions. Furthermore, our system demonstrates real-time processing capability with an end-to-end latency of approximately 22 ms on a standard smartphone. These findings highlight the practical feasibility of camera-only hover sensing and suggest a wide range of touchless interaction applications, enabling touchless interaction when touch is undesirable, pre-touch UI adaptation, and accessibility support on commodity mobile devices.2026YNYudai Nakamura et al.Keio UniversityMulti-Touch Interaction TechniquesMobile App User ExperienceMobile Accessibility DesignIUI
DuoTouch: Passive Two-Footprint Attachments Using Binary Sequences to Extend Touch InteractionDuoTouch is a passive attachment for capacitive touch panels that adds tangible input while minimizing content occlusion and loss of input area. It uses two contact footprints and two traces to encode motion as binary sequences and runs on unmodified devices through standard touch APIs. We present two configurations with paired decoders: an aligned configuration that maps fixed-length codes to discrete commands and a phase-shifted configuration that estimates direction and distance from relative timing. To characterize the system's reliability, we derive a sampling-limited bound that links actuation speed, internal trace width, and device touch sampling rate. Through technical evaluations on a smartphone and a touchpad, we report performance metrics that describe the relationship between these parameters and decoding accuracy. Finally, we demonstrate the versatility of DuoTouch by embedding the mechanism into various form factors, including a hand strap, a phone ring holder, and touchpad add-ons.2026KIKaori Ikematsu et al.LY CorporationTangible User Interface DesignPhysical-Digital Hybrid InteractionMulti-Touch Interaction TechniquesCHI
Amplifying Trigeminal Flavour: Enhancing Spiciness and Coolness by Electrical and Olfactory StimulationDigital flavour modulation represents a significant research challenge within Human-Food Interaction. Previous work has focused on modulating basic tastes, while the modulation of trigeminal sensations such as spiciness and coolness remains underexplored. Spicy and cool substances contribute substantially to the sensory appeal of food and beverages; however, their overconsumption can have adverse health effects. To enhance these trigeminal flavours without chemical additives, this study proposes an integrated multimodal approach combining electrical tongue stimulation with congruent olfactory stimuli. Unlike unimodal methods, our approach leverages the interaction between direct neural stimulation and olfactory integration to selectively modulate distinct spiciness and coolness perception. Psychophysical experiments demonstrated that our method significantly enhanced perceived coolness through combination of electrical tongue stimulation and lemon odor, and significantly enhanced perceived spiciness of a spicy solution by electrical tongue stimulation. These findings suggest that our method expands design space for digital flavour modulation and contributes to healthier and more enriched eating and drinking experiences.2026MOMasaki Ohno et al.The University of TokyoOlfactory Display & Smell InteractionGustatory Interface & Electronic TongueMultisensory Fusion ExperienceCHI
FoodSkin: Fabricating Edible Gold Leaf Circuits on Food SurfacesWe present FoodSkin, a technique for adding interactive elements to foods by implementing edible circuits on the surface of the food. The circuit is easily fabricated using commercially available materials. Existing approaches to enhance the eating experience, such as presenting an electrical taste by making food part of an electronic circuit, are challenging to apply to foods with low water content due to their low conductivity. Our technique enables the integration of dry foods into an electronic circuit and provides displaying (e.g., smell or taste) and sensing (e.g., eating activity) functionalities. We describe our fabrication technique with a library of food materials that we can utilize, evaluate the conductivity and adhesion of the gold-leaf traces, introduce demonstrative applications, and conclude with a workshop we conducted to evaluate the accessibility of our technique. FoodSkin enriches the design space for the computer- augmented eating experience by enabling the digital fabrication of electronics on versatile materials, surfaces, and shapes of foods.2024KKKunihiro Kato et al.Tokyo University of TechnologyDesktop 3D Printing & Personal FabricationFood Culture & Food InteractionCHI
Acoustic+Pose: Adding Input Modality to Smartphones with Near-Surface Hand-Pose Recognition using Acoustic SurfaceTo achieve mid-air interactions for smartphones, acoustic-sensing, a technique using the built-in speaker and microphone of smart- phones, is promising. However, detecting hand poses on the near- surface of touchscreens remains challenging due to the arrange- ment of the built-in speaker and microphone. To address this, we present Acoustic+Pose, a novel approach for combining conven- tional touch interactions with near-surface hand-pose estimation to enable a wide range of interactions. We focused on smartphones incorporating Acoustic Surface, a technology that vibrates the en- tire smartphone screen to emit sound over a wide area. We used this technology to extend the input space to the near surface of touchscreens. We trained machine-learning models to recognize hand poses in the near-surface area and demonstrated interaction techniques to use the recognized poses for a new modality of smart- phone input. Through an evaluation, we confirmed that the trained models recognized 10 hand poses with 90.2% accuracy.2023KKKunihiro Kato et al.Hand Gesture RecognitionUbiComp
Detecting Thumb-Posture for One-handed Interactions with Smartphone using Acoustic SensingThis paper presents a novel approach for expanding one-handed interactions using the thumb positioned above the smartphone screen. Our approach is based on acoustic sensing, a technique for leveraging the built-in speaker and microphone of the smartphone without requiring additional sensors or attachments. We explored the feasibility of our approach on smartphones with the conventional speaker and microphone arrangement and investigated the enhancement of recognition accuracy by using smartphones equipped with Acoustic Surface, which is a technology enabling the entire screen to vibrate and emit sound over a wider area and installed in several commercial smartphones such as LG G8 ThinQ and Huawei P30 Pro. We focused on classifying 12 different thumb postures and developed models that achieve prediction accuracies of 78.6% (conventional smartphone) and 87.0% (Acoustic Surface).2023KKKunihiro Kato et al.Head-Up Display (HUD) & Advanced Driver Assistance Systems (ADAS)Hand Gesture RecognitionUbiComp
ReflecTouch: Detecting Grasp Posture of Smartphone Using Corneal Reflection ImagesBy sensing how a user is holding a smartphone, adaptive user interfaces are possible such as those that automatically switch the displayed content and position of graphical user interface (GUI) components following how the phone is being held. We propose ReflecTouch, a novel method for detecting how a smartphone is being held by capturing images of the smartphone screen reflected on the cornea with a built-in front camera. In these images, the areas where the user places their fingers on the screen appear as shadows, which makes it possible to estimate the grasp posture. Since most smartphones have a front camera, this method can be used regardless of the device model; in addition, no additional sensor or hardware is required. We conducted data collection experiments to verify the classification accuracy of the proposed method for six different grasp postures, and the accuracy was 85%.2022XZXiang Zhang et al.Keio UniversityHuman Pose & Activity RecognitionPrototyping & User TestingCHI
LightTouch Gadgets: Extending Interactions on Capacitive Touchscreens by Converting Light Emission to Touch InputsWe present LightTouch, a 3D-printed passive gadget to enhance touch interactions on unmodified capacitive touchscreens. The LightTouch gadgets simulate finger operations such as tapping, swiping, and multi-touch gestures by means of conductive materials and light-dependent resistors (LDR) embedded in the object. The touchscreen emits visible light and the LDR senses the level of this light, which changes its resistance value. By controlling the screen brightness, it intentionally connects or disconnects the path between the GND and the touchscreen, thus allowing the touch inputs to be controlled. In contrast to conventional physical extensions for touchscreens, our technique requires neither continuous finger contact on the conductive part nor the use of batteries. As such, it opens up new possibilities for touchscreen interactions beyond the simple automation of touch inputs, such as establishing a communication channel between devices, enhancing the trackability of tangibles, and inter-application operations.2021KIKaori Ikematsu et al.Yahoo Japan CorporationCircuit Making & Hardware PrototypingCHI