Acoustic+Pose: Adding Input Modality to Smartphones with Near-Surface Hand-Pose Recognition using Acoustic Surface

To achieve mid-air interactions for smartphones, acoustic-sensing, a technique using the built-in speaker and microphone of smart- phones, is promising. However, detecting hand poses on the near- surface of touchscreens remains challenging due to the arrange- ment of the built-in speaker and microphone. To address this, we present Acoustic+Pose, a novel approach for combining conven- tional touch interactions with near-surface hand-pose estimation to enable a wide range of interactions. We focused on smartphones incorporating Acoustic Surface, a technology that vibrates the en- tire smartphone screen to emit sound over a wide area. We used this technology to extend the input space to the near surface of touchscreens. We trained machine-learning models to recognize hand poses in the near-surface area and demonstrated interaction techniques to use the recognized poses for a new modality of smart- phone input. Through an evaluation, we confirmed that the trained models recognized 10 hand poses with 90.2% accuracy.