Generative Muscle Stimulation: Providing Users with Physical Assistance by Constraining Multimodal-AI with Embodied Knowledge

Paper Title

Generative Muscle Stimulation: Providing Users with Physical Assistance by Constraining Multimodal-AI with Embodied Knowledge

Publication Info

• Topic area: Electrical muscle stimulation (EMS) for physical assistance using multimodal AI.
• Keywords: Electrical muscle stimulation, procedural knowledge, multimodal AI, embodied AI, physical assistance, joint-limits, user study, context-aware systems, haptic feedback, interactive systems.

Background and Problem

• Problem / challenge: Existing EMS-based systems are highly specialized, offering fixed and non-contextual instructions that cannot adapt to new tasks or user contexts.
• Significance: A more general-purpose EMS system could enable broader applications, assisting users in unfamiliar physical tasks without requiring task-specific programming.
• Motivation and related work: Prior work in EMS systems, such as Affordance++, demonstrated task-specific assistance but lacked flexibility and contextual awareness. Multimodal AI and vision-language models (VLMs) have shown potential for contextual reasoning, but they are not directly applicable to EMS due to biomechanical constraints and the need for precise muscle stimulation.

Solution

• Proposed approach: A generative EMS system that uses multimodal AI (e.g., computer vision, large language models) constrained by embodied knowledge (e.g., joint-limits, kinematics) to generate context-aware muscle stimulation instructions.
• Novelty:
  1. Introduced a general-purpose EMS system capable of generating task-specific instructions in real-time.
  2. Integrated multimodal AI with EMS-specific constraints to ensure biomechanical feasibility.
  3. Demonstrated recovery from system errors and user adaptability through a user study.
  4. Open-sourced the system and datasets to accelerate research in EMS-based assistance.
• Procedure and key techniques:
  1. Gather user inputs: spoken requests, POV images, and body pose data.
  2. Use multimodal AI to generate textual task instructions.
  3. Translate textual instructions into movement instructions considering body pose and EMS constraints.
  4. Constrain generated instructions using biomechanical knowledge (e.g., joint-limits, kinematic chain).
  5. Deliver EMS-based physical assistance to users.

Results

• Concrete findings:
  • Technical evaluation showed the system achieved the lowest average edit distance (15.4 weighted, 10 unweighted) compared to ablated versions and a naïve VLM baseline.
  • User study demonstrated a 92% task success rate, with participants recovering from errors in 28% of trials without re-prompting and in 65% of trials by re-prompting.
  • Participants rated their understanding of EMS instructions at 5.8/7 on average.
• Advantage over baselines:
  • Outperformed ablated versions and naïve VLM in generating biomechanically valid and contextually appropriate EMS instructions.
  • Demonstrated flexibility in assisting unfamiliar tasks without pre-programmed instructions.
• Experiments / evaluation:
  • Ablation study: Compared full system, ablated versions, and naïve VLM across 12 tasks using a modified Levenshtein distance metric.
  • User study: Observed participants’ responses to correct and intentionally erroneous EMS instructions across six tasks.
  • Tasks included opening a tilt-turn window, using a disposable camera, and operating a magnetic sweeper.
• Limitations and future work:
  • System latency (~23.6s) due to reliance on cloud-based LLMs.
  • Limited scope of examples and EMS practicality (e.g., electrode placement, calibration).
  • Future work includes improving AI reasoning, EMS hardware, and exploring multimodal combinations (e.g., visuals, audio) for skill acquisition.

Summary

This paper introduces a novel EMS system that generates context-aware muscle stimulation instructions using multimodal AI constrained by biomechanical knowledge. The system demonstrated flexibility in assisting unfamiliar tasks, outperforming baselines in technical evaluations, and enabling users to recover from errors in a user study. While current limitations include latency and EMS practicality, the approach lays the groundwork for general-purpose EMS systems and advances in embodied AI for physical assistance.