ReTouche: Embodied Representations for Self-Guided Piano Learning

Paper Title

ReTouche: Embodied Representations for Self-Guided Piano Learning

Publication Info

• Topic area: Human-Computer Interaction (HCI) for music learning
• Keywords: Embodied representations, self-guided learning, piano pedagogy, interactive systems, projection mapping, metacognitive tools, Disklavier, music education, gesture-based learning, HCI.

Background and Problem

• Problem / challenge: Traditional piano learning tools like sheet music, recordings, and YouTube tutorials are symbolic or disembodied, requiring learners to mentally translate information into physical movements. This creates challenges for beginners, including difficulty in connecting gesture, sound, and action.
• Significance: Addressing these challenges can make piano learning more accessible, engaging, and effective, particularly for self-guided learners who lack regular access to teachers.
• Motivation and related work: Prior HCI systems have focused on performance metrics (e.g., accuracy, speed) or gamification but often neglect embodied and expressive dimensions of learning. Systems like MirrorFugue have explored embodied representations but remain limited to performance viewing rather than active learning contexts.

Solution

• Proposed approach: ReTouche, an interactive system that integrates embodied representations (projected hands, moving keys, falling notes) with practice-control features (segmentation, speed control, recording) to support self-guided piano learning.
• Novelty:
  1. Introduction of embodied representations (situated, sensorimotor, social) for piano learning.
  2. Integration of metacognitive practice-control features for self-regulated learning.
  3. Adaptation of YouTube tutorials into interactive, embodied learning experiences.
  4. Empirical evaluation of embodied representations through qualitative studies.
• Procedure and key techniques:
  1. Augment a Yamaha Disklavier with projection mapping and synchronized MIDI playback.
  2. Process YouTube tutorials to extract MIDI data and calibrate projections for the physical keyboard.
  3. Implement features like adjustable playback speed, sectional looping, and recording with synchronized visualizations.
  4. Conduct three studies: comparative structured observation (n=18), longitudinal autoethnography (n=3), and a focus group with piano teachers (n=4).

Results

• Concrete findings:
  • Embodied representations (e.g., projected hands, moving keys) improved learners’ orientation, mimicry, and multimodal engagement.
  • Practice-control features (e.g., speed slider, segmentation) supported incremental progress and self-regulation.
  • Learners developed personalized strategies over time and reported increased motivation and confidence.
• Advantage over baselines:
  • Compared to YouTube, ReTouche reduced frustration, improved alignment between gesture and sound, and fostered a sense of presence and guidance.
  • Teachers saw potential for ReTouche to complement traditional pedagogy, particularly for technique and expressive phrasing.
• Experiments / evaluation:
  1. Comparative structured observation study (n=18): Compared ReTouche with YouTube, focusing on learner behavior, strategies, and emotional experiences.
  2. Longitudinal autoethnography study (n=3): Examined evolving usage patterns, personalization, and social dynamics over two weeks.
  3. Focus group with teachers (n=4): Explored pedagogical value, limitations, and potential extensions of ReTouche.
• Limitations and future work:
  • Absence of traditional notation limited learners’ understanding of musical structure and phrasing.
  • Falling notes representation was criticized for lacking long-term pedagogical value.
  • Future work could integrate symbolic and embodied representations, explore quantitative learning trajectories, and extend to other embodied skill domains.

Summary

ReTouche introduces embodied representations (projected hands, moving keys, falling notes) and metacognitive practice tools (segmentation, speed control, recording) to enhance self-guided piano learning. Empirical studies showed that ReTouche improved learners’ engagement, reduced frustration, and supported diverse learning strategies compared to YouTube. Teachers recognized its potential for technique and expressive phrasing but highlighted the need for score integration. The system demonstrates how embodied representations can bridge perception, action, and reflection, with implications for broader applications in embodied skill learning.