PianoBand: A Multimodal Wristband Interface for Portable Piano Interaction

Paper Title

PianoBand: A Multimodal Wristband Interface for Portable Piano Interaction

Publication Info

• Topic area: Portable musical interfaces for piano interaction.
• Keywords: PianoBand, wristband interface, IMU sensing, vision-based interaction, portable piano systems, fiducial markers, MIDI controller, music education, dynamic expressivity, ergonomic design.

Background and Problem

• Problem / challenge: Traditional pianos are non-portable, limiting accessibility and creativity. Existing portable solutions, such as roll-up pianos and XR-based systems, suffer from poor interaction quality, limited range, occlusion, and unreliable dynamics detection.
• Significance: Portable piano systems can expand musical practice and creativity to everyday contexts, making music education and performance more accessible.
• Motivation and related work: Vision-based systems with external cameras and hardware-centric solutions have been explored but face constraints such as fixed placement, limited octave range, and high latency. Wrist-worn devices offer promising precision and spatial detail but have not yet been applied to piano interaction with sufficient fidelity.

Solution

• Proposed approach: PianoBand, a multimodal wristband system combining an IMU, an under-wrist RGB camera, and a fiducial-augmented printed keyboard sheet for portable piano interaction.
• Novelty:
  1. Integration of IMU sensing and under-wrist vision for accurate tap detection and fingertip localization.
  2. Lightweight multimodal pipeline enabling real-time piano interaction with dynamic velocity and articulation techniques.
  3. Extensible design supporting MIDI output and customizable keyboard layouts.
• Procedure and key techniques:
  • IMU-based tap detection using a 1D CNN.
  • Vision-based fingertip regression with a multi-task CNN for contact classification and heatmap localization.
  • Key mapping via fiducial marker segmentation and white-black classification.
  • Two usage modes: Basic Mode for efficient note triggering and Professional Mode for advanced articulation techniques.

Results

• Concrete findings:
  • Tap detection achieved 99.9% accuracy in binary classification and 98.7% accuracy in multi-class settings.
  • Fingertip regression attained an average pixel error of 8.90 pixels (0.55% relative error).
  • Online evaluation reported 98.0% tap detection accuracy and 96.3% fingertip regression accuracy, with end-to-end latency averaging 26.38 ms.
• Advantage over baselines:
  • Comparable note accuracy to roll-up pianos (95.5% vs. 96.9%) and significantly higher than XR pianos (76.1%).
  • Faster learning time for novices (173 s vs. 356 s for roll-up pianos).
  • Higher user ratings for portability (6.3/7), expressivity (5.6/7), and extensibility (5.8/7).
• Experiments / evaluation:
  • Mixed-methods study with 15 participants (5 professionals, 10 novices) comparing PianoBand to roll-up and XR pianos.
  • Expert interviews with a senior piano teacher and a professional composer.
  • Robustness tests under varying illumination and motion blur conditions.
• Limitations and future work:
  • Ergonomic constraints in wristband design, including discomfort during prolonged use.
  • Performance degradation under rapid tempos (>110 BPM) due to motion blur.
  • Limited evaluation of two-handed and advanced articulation techniques.
  • Need for larger-scale, long-term user studies.

Summary

PianoBand introduces a wrist-worn multimodal system for portable piano interaction, integrating IMU sensing, under-wrist vision, and a printed keyboard sheet. It achieves high accuracy in tap detection and fingertip localization, supports dynamic expressivity, and offers extensibility as a MIDI-compatible interface. User studies and expert feedback validate its usability, portability, and potential applications in practice, education, and creative workflows. Future work will focus on improving ergonomics, robustness, and advanced functionalities to further enhance its applicability as a lightweight, adaptive musical interface.