Rob2HanD: LLM-Driven Robotic Arm for IMU Interaction Dataset Generation

Paper Title

Rob2HanD: LLM-Driven Robotic Arm for IMU Interaction Dataset Generation

Publication Info

• Topic area: IMU-based human-computer interaction and dataset generation using robotic arms and LLMs.
• Keywords: IMU, robotic arm, dataset generation, large language models, human-computer interaction, machine learning, motion analysis, fine-grained interaction, sensor-based HAR, trajectory generation.

Background and Problem

• Problem / challenge: Existing IMU-based interaction datasets rely heavily on real user data collection, which is time-consuming, labor-intensive, and lacks diversity and personalization. Synthetic datasets often fail to capture fine-grained motions or realistic noise characteristics.
• Significance: Addressing these challenges is crucial for improving the robustness, generalizability, and efficiency of machine learning models in human-computer interaction applications.
• Motivation and related work: Prior work has explored robotic arms for dataset generation and LLMs for motion understanding, but these approaches either lack bio-fidelity or struggle with fine-grained interaction scenarios. This paper aims to bridge these gaps by integrating LLMs and robotic arms to generate large-scale, personalized, and realistic datasets.

Solution

• Proposed approach: Rob2HanD, a tool combining robotic arms, large language models (LLMs), and human motion data to generate large-scale IMU datasets for machine learning.
• Novelty:
  1. A framework for integrating LLMs with robotic arms to generate diverse, realistic, and large-scale datasets.
  2. Detailed analysis of human hand movement features to inform robotic arm motion.
  3. LLM-based regulation of robotic arm movements to improve bio-fidelity in fine-grained interaction scenarios.
  4. Tools for rapid trajectory generation in both zero-shot and few-shot scenarios.
• Procedure and key techniques:
  1. Conduct user studies to collect real human motion data and analyze kinematic features.
  2. Use LLMs to regulate robotic arm movements based on prior knowledge derived from human data.
  3. Generate large-scale datasets by simulating human-like motions with a robotic arm equipped with IMU sensors.
  4. Incorporate trajectory generation algorithms and LLM-based optimization to ensure dataset quality and diversity.

Results

• Concrete findings:
  • Models trained on Rob2HanD-generated datasets achieved 85.13% accuracy on real human datasets, improving to 96.14% with minimal real data integration.
  • Rob2HanD reduced data collection time by 54.1% compared to human-based methods.
  • LLM-based regulation improved dataset quality, with a 2.57% accuracy increase over unregulated datasets.
• Advantage over baselines:
  • Outperformed IMUGPT in accuracy (96.14% vs. 82.68% with real data augmentation).
  • Retained 85.72% of the performance of fully human-collected datasets, with significant efficiency gains.
• Experiments / evaluation:
  • Compared datasets generated by Rob2HanD with real human data and IMUGPT.
  • Evaluated models using CNN-LSTM architecture with 5-fold cross-validation.
  • Conducted experiments on five gestures (four dynamic, one static) with both robotic arm and human participants.
• Limitations and future work:
  • Limited simulation capability for fingertip joint movements and long-range full-body actions.
  • Hardware constraints on robotic arm end-effectors.
  • Future plans include integrating high-degree-of-freedom fingertip simulation, expanding motion features, and incorporating multimodal trajectory generation methods.

Summary

Rob2HanD is a novel framework that integrates robotic arms and large language models (LLMs) to generate large-scale, realistic IMU datasets for machine learning in human-computer interaction. By analyzing human motion data and leveraging LLMs for robotic arm regulation, Rob2HanD achieves high dataset quality and diversity while significantly reducing data collection time. Experiments demonstrate that models trained on Rob2HanD-generated datasets perform comparably to those trained on human-collected data, with strong generalization capabilities. This tool has broad applications in sensor-based human activity recognition, motion simulation, and interactive systems, with potential for further enhancements in hardware and trajectory generation techniques.