FIP: Endowing Robust Motion Capture on Daily Garment by Fusing Flex and Inertial Sensors

Research Background and Problem

• Problems and Challenges: Current motion capture (MoCap) technologies face various challenges, particularly the issue of decreased sensor accuracy when worn under loose-fitting clothing. Loose clothing causes sensor displacement, leading to significant joint tracking errors. Specifically, there are two types of displacement: primary displacement during initial wear and real-time displacement during actual movement.
• Significance: Achieving accurate and versatile motion capture technology has broad applications in motion analysis, rehabilitation, and metaverse interactions. However, existing technologies that rely solely on IMUs (Inertial Measurement Units) or flexible sensors are inadequate, failing to balance precision and user comfort.
• Research Motivation: The authors propose a multimodal approach that integrates IMUs and flexible sensors to overcome the limitations of single-sensor systems. This approach also addresses sensor displacement caused by loose clothing, significantly improving motion capture accuracy.

Solution

Methodology and Innovations

• System Design: The authors introduce the Flexible Inertial Poser (FIP), a novel motion capture system for loose clothing that integrates four IMUs and two flexible sensors to capture upper limb movements.
• Key Technical Innovations:
  1. Displacement Latent Diffusion Model (DLDM):
     • Synthesizes real-time IMU displacement data, enabling the model to handle diverse data distributions during training and enhance robustness.
  2. Physics-Informed Calibrator (PIC):
     • Corrects primary displacement of flexible sensors by calibrating actual motion angles through simple elbow flexion.
  3. Pose Fusion Predictor (PFP):
     • A multimodal data fusion technique that combines readings from IMUs and flexible sensors to achieve more accurate pose estimation.

Implementation Steps

1. Data Collection:
   • Training data is generated using simulated data (IMU signals under tight and loose clothing conditions, as well as their differences).
   • Real-world data is collected by comparing with tight-fitting devices (e.g., Perception Neuron 3 system).
2. Data Processing and Model Construction:
   • DLDM generates IMU displacement data using a Variational Autoencoder (VAE) and employs Diffusion Models to produce high-quality and diverse displacement data.
   • Flexible sensors are corrected for primary displacement using PIC.
   • PFP utilizes a multi-layer Long Short-Term Memory (LSTM) network to combine multimodal sensor data for pose estimation.
3. Real-Time Operation:
   • After calibration, the system operates at a frequency of 60 Hz, enabling smooth real-time capture.

Research Outcomes

Key Results

• Accuracy Improvement:
  • Total angular error reduced by 19.5%, elbow error reduced by 26.4%, and joint position error reduced by 30.1%.
  • The inclusion of flexible sensors significantly reduced elbow tracking errors.
• Advantages Over Existing Technologies:
  • Under loose clothing conditions, FIP outperforms all current IMU-based real-time motion capture methods, including DIP, PIP, and LIP.
  • FIP strikes a balance between comfort and accuracy, representing an innovative solution for motion capture under loose clothing.
• Experimental Results:
  • Extensive experimental validation demonstrates that FIP exhibits strong generalization across various motion patterns and user body types.
  • The system operates at a stable frequency of 60 Hz with high reliability and real-time performance.

Limitations and Future Directions

• Hardware Limitations:
  • The impact of different clothing sizes and fabrics on capture performance requires further investigation.
  • The system's performance on low-power embedded devices has not yet been validated.
• Algorithm Improvements:
  • While PIC addresses initial displacement correction for flexible sensors, correcting large-scale real-time sensor displacement (e.g., during intense movements) remains a technical challenge.
  • Incorporating clothing characteristics (e.g., tailoring, fabric) and user body types into the modeling algorithm could further enhance the system's generalizability and accuracy.

Conclusion

The authors propose a real-time, accurate motion capture method that integrates IMUs and flexible sensors to achieve robust motion capture under loose clothing. The study demonstrates the potential applications of this technology in fields such as the metaverse, rehabilitation, and fitness, while also pioneering new directions for motion capture research under loose clothing conditions. This research not only advances the state of the art but also lays a foundation for further exploration of sensor fusion potential.