AgentCoach: LLM-Based Adaptive Coaching Feedback for Motor Skill Learning

Paper Title

AgentCoach: LLM-Based Adaptive Coaching Feedback for Motor Skill Learning

Publication Info

• Topic area: Adaptive feedback systems for motor skill learning using large language models (LLMs).
• Keywords: Motor skill learning, adaptive feedback, large language models, coaching points, pose estimation, multimodal feedback, sports training, tutorial videos, personalized coaching, biomechanics.

Background and Problem

• Problem / challenge: Existing motor learning systems provide limited, non-adaptive feedback that lacks the nuanced, personalized guidance of human coaches. They often focus on visual posture comparison without connecting errors to actionable coaching principles or adapting feedback to learner progress.
• Significance: High-quality, personalized feedback is critical for effective motor skill acquisition, but access to human coaching is often limited by cost, geography, and availability.
• Motivation and related work: Prior systems rely on pose estimation and visual feedback but fail to provide semantically meaningful, adaptive coaching. Advances in LLMs and vision-language models (VLMs) offer the potential to bridge this gap by extracting coaching points (CPs) from tutorial videos and generating context-aware feedback.

Solution

• Proposed approach: AgentCoach, a multimodal LLM-powered system, provides adaptive, CP-based feedback by linking high-level coaching principles to measurable kinematic parameters extracted from tutorial videos.
• Novelty:
  1. A CP-to-parameter mapping library that bridges high-level instructions to low-level pose-based measures.
  2. A multimodal feedback system combining visual diagnostics with adaptive verbal coaching based on CP-wise evaluation and performance history.
  3. A history-aware progression policy that adjusts feedback type and timing to learner progress.
• Procedure and key techniques:
  1. Extract CPs and reference motion segments from tutorial videos using Gemini and MediaPipe.
  2. Map CPs to measurable parameters (e.g., joint angles, limb orientations) using a predefined taxonomy.
  3. Evaluate user motion against reference parameters and track CP-wise performance history.
  4. Deliver adaptive feedback combining visual overlays and verbal cues, generated via LLMs and synthesized with text-to-speech (TTS).

Results

• Concrete findings:
  • CP extraction achieved an F1 score of 91.57 (macro) and 90.07 (micro) in a few-shot setup.
  • CP-to-parameter mapping reached exact match scores of 93.69% (macro) and 94.06% (micro).
  • User study participants rated AgentCoach’s feedback as more comprehensible (AVG = 4.12), actionable (AVG = 4.21), and confidence-boosting (AVG = 4.00) compared to baselines.
  • Learning accuracy improved most under AgentCoach (C3), reaching ≥95% by the 10th repetition.
• Advantage over baselines:
  • Outperformed visual-only (C1) and diagnostic verbal feedback (C2) in clarity, actionability, and learning outcomes.
  • Reduced mental demand (AVG = 2.62) compared to C1 (AVG = 3.51) and C2 (AVG = 3.50).
• Experiments / evaluation:
  • Validated CP extraction and parameter mapping using 55 tutorial videos.
  • Conducted a two-phase user study with 24 participants practicing six motor skills under three feedback conditions (C1–C3).
  • Metrics included precision, recall, F1, learning accuracy, and subjective ratings on usability and experience.
• Limitations and future work:
  • Limited to postural feedback; cannot address muscle activation or weight distribution.
  • Requires clean reference video segments; struggles with complex, continuous, or composite skills.
  • Fixed thresholds for misalignment may not generalize across diverse users.
  • Pose estimation errors can degrade feedback accuracy.

Summary

AgentCoach is an LLM-powered system that transforms tutorial videos into adaptive, CP-based coaching for motor skill learning. By combining visual diagnostics with history-aware verbal feedback, it bridges high-level coaching principles with low-level pose analysis. User studies demonstrate its effectiveness in enhancing learning outcomes, user confidence, and feedback clarity compared to non-adaptive baselines. While currently limited to postural feedback for discrete skills, future work aims to expand its capabilities to address non-postural cues, complex routines, and personalized thresholds. AgentCoach represents a significant step toward scalable, personalized motor skill training.