SoleCoach: Sole Pressure and IMU-based MLLMs for Skill Coaching

Paper Title

SoleCoach: Sole Pressure and IMU-based MLLMs for Skill Coaching

Publication Info

• Topic area: AI-driven coaching systems for sports training using wearable sensors.
• Keywords: sports coaching, insole sensors, IMU, multimodal large language models, alpine skiing, skill assessment, feedback generation, foot pressure, posture estimation, user study.

Background and Problem

• Problem / challenge: Existing coaching systems rely heavily on external sensors such as cameras or motion-capture devices, which are impractical for outdoor sports like skiing. These systems lack the ability to provide comprehensive, actionable feedback based solely on wearable sensors.
• Significance: Effective coaching is critical for skill acquisition in sports, but access to expert coaches is limited. A portable, sensor-based system could democratize coaching, especially in outdoor environments.
• Motivation and related work: Prior systems use pose-based methods or video analysis for feedback, but these approaches are unsuitable for large-scale movements in outdoor sports. While insole sensors have been explored for motion analysis, they have not been fully utilized for generating coaching feedback. This paper addresses the gap by leveraging insole sensor data and multimodal large language models (MLLMs).

Solution

• Proposed approach: SoleCoach, a coaching system that generates feedback using only insole sensor data (foot pressure and IMUs) combined with MLLMs to interpret motion patterns and provide actionable feedback.
• Novelty:
  1. Development of a coaching system that operates without external cameras or motion capture, relying solely on insole sensors.
  2. Creation of a new alpine skiing dataset with multimodal data (foot pressure, IMUs, 3D poses) and 387 expert coaching comments.
  3. Introduction of a body-part-based evaluation metric (AutoBCE) to assess coaching quality.
  4. Demonstration of the system’s effectiveness through user studies and quantitative evaluations.
• Procedure and key techniques:
  • Train a model to estimate skier posture from insole sensor data.
  • Use a multimodal large language model (Qwen2.5-7B-Instruct) to generate coaching feedback by integrating posture and foot pressure data.
  • Develop a synthetic dataset to align foot pressure and textual descriptions, enhancing the model’s interpretability.
  • Evaluate the system using both automated metrics (e.g., BLEU, AutoBCE) and human studies.

Results

• Concrete findings:
  • SoleCoach achieved the highest scores across all evaluation metrics, including BLEU (5.18), METEOR (26.7), ROUGE-L (28.1), and AutoBCE-F1 (0.452).
  • Human evaluations showed SoleCoach’s coaching quality was comparable to expert feedback, with a selection rate of 45.1% against human coaches.
  • User study participants rated SoleCoach highly for identifying correction points (6.1/7) and increasing motivation (6.7/7).
• Advantage over baselines:
  • Outperformed existing models like ExpertAF and Qwen2.5-VL in generating ski-specific coaching.
  • Demonstrated that insole sensor data alone can surpass externally captured posture data in providing actionable feedback.
• Experiments / evaluation:
  • Quantitative evaluation using metrics like BLEU, METEOR, and AutoBCE.
  • Ablation studies to assess the contributions of input modalities (foot pressure, estimated posture) and synthetic datasets.
  • User study with six novice-to-intermediate skiers comparing three conditions: IMU-based posture visualization, insole-based visualization, and SoleCoach with coaching feedback.
• Limitations and future work:
  • Limited ability to capture environmental context, such as pole usage or course layout.
  • Challenges in prioritizing feedback timing and addressing subtle upper-body movements.
  • Need for further validation with advanced athletes and long-term studies.

Summary

SoleCoach is a novel coaching system that uses insole sensor data and MLLMs to provide actionable feedback for alpine skiing. It eliminates the need for external cameras or motion capture, making it suitable for outdoor sports. The system demonstrated superior performance over baselines in generating expert-level feedback and enhancing user motivation. While the approach shows promise, future work should address environmental context, feedback prioritization, and broader applicability across skill levels and sports.