WheelPose: Data Synthesis Techniques to Improve Pose Estimation Performance on Wheelchair Users

Title of the Paper

WheelPose: Data Synthesis Techniques to Improve Pose Estimation Performance on Wheelchair Users

Paper Information

• Subject Area: Data synthesis, AI fairness, accessibility systems
• Keywords: Data synthesis, AI fairness, wheelchair users, human pose estimation, human motion modeling, accessibility technology, domain randomization, computer vision, Unity engine, human model

Research Background and Problem

• Problems or Challenges Identified by the Authors:

  • Existing human pose estimation models perform poorly on wheelchair users, primarily due to the lack of representative wheelchair user data in training datasets.
  • Data collection processes are often inaccessible to individuals with disabilities, including challenges in using motion capture devices and risks associated with unsafe postures.
  • Traditional manually annotated datasets (e.g., COCO, ImageNet) are costly and lack diversity, particularly failing to adequately represent disabled populations.

• Why This Problem is Important:

  • The fairness and inclusivity of AI models directly depend on the quality and representativeness of training data. Disabled groups (e.g., wheelchair users) are systematically overlooked in many AI system applications.
  • Pose estimation for wheelchair users has significant value, such as detecting sitting posture, predicting health risks (e.g., pressure ulcers), and analyzing emotions.

• Research Motivation and Related Work:

  • The authors drew on prior work on synthetic data generation and domain randomization, finding that synthetic data can compensate for gaps in real-world datasets.
  • Driven by the need for AI fairness and diverse data acquisition, the authors developed a synthetic data generation framework to improve the performance of AI models related to wheelchair users.

Solution

• Methods or Solutions Proposed by the Authors:

  • The authors proposed a data synthesis framework called WheelPose, which generates synthetic wheelchair user data based on a Unity simulation environment. This includes steps such as human motion generation, data filtering, and scene simulation.
  • The framework supports the generation of diverse data through domain randomization (e.g., randomizing backgrounds, lighting, and human models) and outputs pose annotations in COCO 17 keypoint format.

• Innovative Aspects of the Solution:

  • Created the first fully annotated dataset for wheelchair users, addressing the lack of representation of disabled populations in existing datasets.
  • Integrated automated motion generation and text-driven motion generation models (e.g., Text2Motion), providing a configurable and extensible data simulation environment.

• Implementation Steps and Key Technologies:

  1. Human Pose Generation: Generate animation keyframes for wheelchair users using existing motion capture data (HumanML3D) and text-driven motion generation models (Text2Motion).
  2. Human Model and Scene Configuration: Use Unity SyntheticHumans to provide diverse human models with customizable appearances, clothing, etc.
  3. Simulation Environment Generation: Use Unity Perception to achieve domain randomization, such as random background images, lighting parameters, and camera positions.
  4. Output Synthetic Data: Generate the final synthetic image dataset with COCO 17 keypoint pose annotations.

Research Outcomes

• Specific Achievements:

  • Constructed a new synthetic dataset containing 70,000 wheelchair user images, each with complete pose annotations.
  • Synthetic data significantly improved the performance of Detectron2 models trained on ImageNet for wheelchair user detection and keypoint estimation.

• Advantages Compared to Existing Solutions:

  • Improved model fairness and inclusivity, particularly excelling in keypoint prediction for self-occlusion scenarios in wheelchair users.
  • Generated data with higher diversity and realism, applicable to real-world tasks such as blind spot detection and obstacle analysis.

• Experimental or Evaluation Results:

  • On the test dataset, compared to ImageNet baseline models, bounding box detection mAP improved by 98%, and keypoint prediction mAP increased by 7.81%.
  • Human evaluation tests showed high realism of the synthetic motion sequences, with model performance further improving after removing motions deemed "unrealistic" by human evaluators.

• Limitations and Future Directions:

  • Limitations:

    • The dataset does not fully cover all types of wheelchair users (e.g., individuals with amputations, dwarfism), necessitating the expansion of more diverse body models in the future.
    • The sample size for human evaluations was relatively small (13 wheelchair users), limiting representativeness.

  • Future Directions:

    • Expand modeling and data synthesis support for other disabled groups (e.g., crutch users, prosthetic users).
    • Incorporate advanced 3D environment generation technologies (e.g., NeRF) to enhance scene realism.
    • Develop more sophisticated motion generation models to capture more natural wheelchair user behaviors.

Through this research, the authors have made a significant contribution to the application of synthetic data generation in accessibility technologies and provided an open tool platform for the research community (Code repository link: https://github.com/hilab-open-source/wheelpose).