Skill-Adaptive Ghost Instructors: Enhancing Retention and Reducing Over-Reliance in VR Piano Learning

Paper Title

Skill-Adaptive Ghost Instructors: Enhancing Retention and Reducing Over-Reliance in VR Piano Learning

Publication Info

• Topic area: Virtual Reality (VR)-based motor-skill learning, specifically piano training.
• Keywords: VR piano training, motor-skill learning, adaptive transparency, ghost instructor, skill retention, learner autonomy, visual guidance, pitch accuracy, timing precision, fingering correctness.

Background and Problem

• Problem / challenge: Existing VR-based music training systems often rely on persistent visual cues, leading to over-reliance, weaker skill retention, and insufficient focus on proper fingering strategies. Fixed-visibility guidance fails to adapt to learner performance, limiting its effectiveness in promoting autonomy and retention.
• Significance: Addressing over-reliance on visual cues is critical for improving skill retention and transfer in motor-skill learning, particularly in tasks requiring spatial precision, timing, and ergonomic control, such as piano playing.
• Motivation and related work: Prior research has shown the potential of VR for motor-skill learning but highlights gaps in adaptive feedback mechanisms. Fixed-visibility systems dominate, while adaptive approaches remain underexplored, particularly for fine-motor tasks like piano training. This paper builds on the need for adaptive guidance that balances instructional support with learner autonomy.

Solution

• Proposed approach: A skill-adaptive ghost instructor system that dynamically adjusts its transparency based on real-time learner performance in pitch accuracy, timing precision, and fingering correctness.
• Novelty:
  1. Introduction of adaptive transparency as a design principle to balance instructional support and learner autonomy in VR-based motor-skill training.
  2. Development of a real-time performance-driven transparency controller that fades guidance during proficient performance and restores it after errors.
  3. Empirical validation of the approach through a controlled user study, demonstrating improved retention, reduced error rates, and lower perceived workload compared to static guidance.
• Procedure and key techniques:
  • A VR piano training system overlays a ghost hand performing target fingerings.
  • Two guidance modes: Static (fixed transparency) and Dynamic (performance-adaptive transparency).
  • Real-time performance metrics (pitch, timing, fingering) drive the ghost hand’s visibility using a weighted performance score and an asymmetric exponential moving average.
  • Evaluation through a within-subject study with 30 participants, measuring objective performance (accuracy, error rates) and subjective workload.

Results

• Concrete findings:
  • Dynamic guidance improved pitch accuracy (99.6% retention vs. 87.8% for Static) and fingering accuracy (96.5% retention vs. 76.7% for Static).
  • Error rates were lower under Dynamic (11.6% retention) compared to Static (19.8% retention).
  • Timing accuracy remained similar across conditions, with no significant differences.
• Advantage over baselines:
  • Dynamic guidance yielded better retention of pitch and fingering accuracy, limited error increases, and reduced perceived workload compared to Static guidance.
  • Participants who trained with Dynamic first showed better skill transfer when switching to a new melody and condition.
• Experiments / evaluation:
  • A within-subject study with 30 participants (15 experienced, 15 inexperienced pianists).
  • Two short piano melodies practiced under Static and Dynamic conditions, with counterbalanced order.
  • Objective metrics: pitch, timing, and fingering accuracy; error rates.
  • Subjective metrics: NASA-TLX workload, Likert-scale evaluations, and open-ended feedback.
• Limitations and future work:
  • Fixed weighting of performance metrics may not suit all learners; future work should explore personalized weighting schemes.
  • Tracking inaccuracies (e.g., ring/pinky fingers) and lack of tactile feedback from a physical keyboard.
  • Limited ecological validity due to simplified tasks (monophonic, short phrases) and short retention intervals.
  • Future directions include bimanual/polyphonic tasks, rhythm aids, and domain-specific validation in sports and rehabilitation.

Summary

This paper introduces a skill-adaptive ghost instructor system for VR piano training, which dynamically adjusts transparency based on learner performance. Compared to static guidance, the adaptive system significantly improved pitch and fingering accuracy, reduced error rates, and enhanced short-term retention, while also lowering perceived workload. The findings highlight the potential of adaptive transparency to balance instructional support and learner autonomy. Applications extend to other motor-skill domains requiring spatial precision and rhythmic control, though further research is needed to address limitations in tracking, personalization, and ecological validity.