PiaMuscle: Improving Piano Skill Acquisition by Cost-effectively Estimating and Visualizing Activities of Miniature Hand Muscles

Research Background and Problem

• Identified Problems or Challenges:

  • Piano performance involves fine finger movements requiring coordination of multiple small hand muscles, yet existing technologies primarily target large muscle activities and lack precise evaluation methods for small muscles.
  • Current surface electromyography (EMG) sensors are expensive, complex, and inflexible, limiting their use in natural environments.
  • Piano learners struggle to accurately assess their muscle exertion through traditional methods like video, leading to inefficiencies in muscle strength control training.

• Importance of the Problem:

  • Precisely understanding and optimizing small muscle activities can enhance piano performance skills, prevent skill degradation, and improve expressiveness.
  • Providing a low-cost method for estimating small muscle activity can be extended to music training and other fine motor skill applications (e.g., esports, handicrafts).

• Research Motivation and Related Work:

  • Although musculoskeletal modeling and data-driven methods exist for estimating large muscle activities, low-cost estimation methods for small muscles remain unresolved.
  • Data collection techniques for natural scenarios, such as video and inertial measurement units (IMU), are more economical than traditional sensors. Thus, the authors explore combining multimodal data (video, piano key data) to replace complex sensors in estimating small muscle activity.

Solution

• Proposed Method or Solution:

  • Developed a system named PiaMuscle that combines multimodal data (video and piano key strike data) with high-precision neural network models to estimate small hand muscle activity.
  • Designed a muscle activity visualization interface to provide performers with intuitive feedback on muscle usage patterns, enabling optimized strength control during piano practice.

• Innovations:

  1. Natural Data Collection and Estimation:
     • Proposed a low-cost muscle activity estimation method based on video and key strike data.
  2. Multimodal Data Integration:
     • Designed "Discrete Electromyography Representation" (DER) and "Discrete Posture Representation" (DPR) modules to extract high-quality discrete features from complex input data (e.g., video sequences).
  3. Muscle Visualization Application:
     • Through PiaMuscle, estimated EMG activity is aligned with video and dynamically annotated differences are provided, helping performers better understand and optimize muscle usage.

• Implementation Steps:

  1. Data Collection:
     • Multimodal data includes: video recording hand movements, key sensors capturing piano key dynamics, and EMG sensors providing real muscle activity data as a training set.
  2. Data Processing:
     • Applied Butterworth filtering for smoothing and noise reduction, combined with maximum voluntary contraction (MVC) normalization to minimize individual differences.
  3. Model Design:
     • Stage One: VQ-VAE generates discrete muscle and posture representations.
     • Stage Two: A Transformer network based on cross-attention mechanisms (EMGFormer) learns the relationship between multimodal inputs and EMG outputs.
  4. Visualization Design:
     • An interactive interface displays hand models and muscle activity, visually presenting differences with reference data to support performers in adjusting muscle strength.

Research Outcomes

• Specific Outcomes:

  • Successfully developed a low-cost muscle activity estimation model combining multimodal inputs (video/key data) and validated its effectiveness and usability through user studies.
  • PiaMuscle demonstrated excellent performance in improving muscle strength control accuracy and provided a new dimension for observing internal information during piano training.

• Comparison with Existing Solutions:

  • Compared to traditional high-cost, high-complexity methods based on EMG sensors, this study offers a more economical and deployable alternative for natural environments.
  • PiaMuscle not only quantifies muscle activity but also enhances user understanding through intuitive difference visualization, which traditional video methods cannot achieve.

• Experimental or Evaluation Results:

  1. Model Performance:
     • In terms of estimation accuracy (MSE), the model combining video and key data significantly outperformed single-modal input models.
  2. User Testing:
     • Five professional pianists participated in the user study. Results showed that compared to traditional video-only methods, PiaMuscle significantly improved muscle activity optimization (average improvement of 10.44%) and key strength control.
  3. User Feedback:
     • The system's usability (SUS score of 73.5) and user experience (UEQ-S scores surpassing video-based solutions) were excellent. Users generally found it trustworthy, intuitive, and helpful for skill enhancement.

• Limitations and Future Directions:

  1. Data and User Scale:
     • The current dataset and participant numbers are limited, involving only six professional pianists. Future work should expand to users of varying skill levels and collect larger, more diverse datasets.
  2. Model Generalization:
     • The model's generalization ability is constrained by the training data distribution. Future exploration of few-shot learning or transfer learning could improve adaptability.
  3. Additional Improvements:
     • Provide more dynamic muscle models to support corrective suggestions or pre-annotate target muscle strength in sheet music.

Conclusion

The PiaMuscle system demonstrates strong potential for estimating and visualizing small muscle activity using low-cost sensing technologies in natural environments. User study results indicate it effectively enhances strength control and skill acquisition in piano performance, while offering valuable insights and development possibilities for other domains requiring fine motor skills.