IMUPoser: Full-Body Pose Estimation using IMUs in Phones, Watches, and Earbuds

Title of the Paper

IMUPoser: Full-Body Pose Estimation using IMUs in Phones, Watches, and Earbuds

Paper Information

• Topic Area: Full-body pose estimation using inertial measurement units (IMUs) in consumer-grade devices
• Keywords: Full-body motion capture, sensors, inertial measurement units, mobile devices, pose estimation, wearable devices, deep learning, data synthesis, real-time systems, healthcare

Research Background and Problem Statement

• Identified Problems or Challenges:

  • Full-body pose tracking is crucial for applications such as virtual reality, gaming, fitness, rehabilitation, and context-aware systems. However, existing high-precision systems (e.g., XSens or Vicon) are expensive and require bulky equipment.
  • IMU data from consumer-grade devices (e.g., smartphones, smartwatches, and earbuds) is sparse, and device configurations change over time, making it challenging to estimate complete human poses from such variable and sparse data.
  • Consumer-grade IMU data is of lower quality, with high noise levels and low frame rates.

• Importance of the Problem: Generating useful pose information from everyday devices that users already own could have a significant impact on health monitoring, motion analysis, and daily activity tracking. This would enable widespread adoption without the need for costly or specialized hardware, greatly lowering the barrier to entry.

• Research Motivation and Related Work:

  • Commercial systems (e.g., XSens, Vicon) and pose capture methods based on external devices (e.g., Kinect, depth cameras) offer high precision but face challenges in terms of cost and portability.
  • Some studies have attempted to capture specific joint movements using IMU sensors or individual devices, but these often require synchronization of multiple high-quality sensors.
  • Currently, there is no real-time, accurate full-body pose estimation solution based on sparse consumer-grade IMU devices.

Proposed Solution

• Proposed Method or Solution:

  • A new system called IMUPoser is introduced, utilizing IMUs in smartphones, smartwatches, and earbuds to achieve real-time full-body pose estimation.
  • The system dynamically detects the location of devices on the body and combines IMU data to estimate human poses. A model architecture was designed to accommodate sparse sensing points and incomplete inputs.
  • The initial model training was based on synthetic IMU data generated from high-resolution motion capture data, followed by fine-tuning and testing using data collected from real consumer devices.

• Innovations:

  • Supports sparse device distribution and dynamic changes, accommodating combinations of 1 to 3 IMU sensor devices.
  • Operates in real-time without requiring specialized hardware, relying entirely on consumer-owned devices.
  • The model design includes robust handling of missing devices and mechanisms for smoothing data loss.
  • Introduces a multi-layer bidirectional LSTM architecture to ensure temporal coherence in predictions while achieving low computational complexity for real-time performance.

• Implementation Steps and Technical Details:

  1. Model Design: Based on a bidirectional LSTM architecture (256-dimensional hidden layers), the model takes device IMU data (3 acceleration axes + 9 rotation matrix) as input and outputs 144 parameters for the SMPL human model pose.
  2. Data Generation and Calibration: IMU pose data was generated using the AMASS dataset, with algorithms simulating device absence. IMU sensors were calibrated using Swift's CoreMotion API.
  3. Model Training: Supervised learning was conducted using PyTorch, with a loss function incorporating MSE errors based on full-body joint positions and rotations.
  4. Inverse Kinematics Optimization: Pose refinement for specific joints (e.g., wrists, shoulders, and head) was achieved using IMU absolute orientation data.
  5. Real-Time System: Implemented within the Apple device ecosystem to enable data streaming and real-time pose estimation, with an average inference time of 26.8 ms.

Research Outcomes

• Specific Results:

  • Average errors on IMUPoser and DIP-IMU datasets were 14.1 cm and 12.1 cm, respectively.
  • With only one device, the average error was 16.27 cm; with three devices, the error reduced to 11.1 cm.
  • On the DIP-IMU dataset, the MPJVE was 12.1 cm, differing by only 3–5 cm from advanced systems requiring more IMUs.

• Advantages Compared to Existing Solutions:

  • Real-time implementation using consumer-grade devices, significantly reducing deployment costs and complexity compared to more specialized or complex hardware (e.g., XSens, Deep Inertial Poser).
  • Flexible adaptation to dynamic changes in device quantity and location, supporting various device combinations.

• Experimental or Evaluation Results:

  • Robust performance across various tasks (e.g., gait, jumping, head movements), particularly for activities with clear motion patterns (e.g., symmetrical movements), even with sparse sensing points.
  • For cross-leg movements compared to no-device configurations, left and right leg trajectories were similar, with no significant error accumulation.
  • Real-time automatic tracking of user devices achieved a position recognition accuracy of over 90%.

• Limitations and Future Directions:

  • Pose inference accuracy for non-sensing parts is limited, making it difficult to handle highly independent local limb movements.
  • Currently does not support dynamic activities (e.g., cycling, rowing) or global displacement estimation.
  • Real-time implementation relies on a single device ecosystem (Apple); future work may explore cross-platform compatibility.
  • The system could be expanded to support additional devices (e.g., smart shoes, wearable rings).