Per Garment Capture and Synthesis for Real-time Virtual Try-on

Title of the Paper

Per Garment Capture and Synthesis for Real-time Virtual Try-on

Paper Information

• Subject Area: Virtual try-on, computer vision, and image synthesis
• Keywords: Virtual try-on, deep image synthesis, image-to-image translation, deformable human model, garment capture, real-time interaction

Research Background and Problem

• Identified Problems or Challenges:
  • Current image-based virtual try-on methods fail to accurately simulate wrinkle variations caused by body shape changes or different interactions (e.g., stretching or grabbing).
  • Conventional virtual try-on technologies struggle to realistically reproduce garment fit and dynamic appearance.
  • Generating try-on effects from a single image limits interaction richness and makes it difficult to synthesize high-quality results.
• Importance of the Problem:
  • With the rise of online shopping (especially during the pandemic), virtual try-on has become a key technology for enhancing the shopping experience, with rapidly growing demand.
  • Higher-quality and more realistic virtual try-on can help consumers make more informed purchasing decisions.
• Research Motivation:
  • To overcome the limitations of existing try-on technologies and provide users with richer interaction methods and more realistic try-on effects in virtual environments.
  • To capture detailed data for each garment to support a personalized try-on experience.

Solution

• Proposed Method or Solution:
  • Introduced a "per garment capture" pipeline, combining a customizable movable mannequin and measurement suit.
  • Leveraged a depth image-to-image translation network (pix2pixHD model) trained on captured images to achieve personalized try-on effects.
  • Utilized a measurement suit to enable real-time interaction through a unified garment representation.
• Innovations:
  • Proposed for the first time a "per garment capture" approach to record detailed garment deformation and interaction effects.
  • Designed a specialized measurement suit texture representation to support more accurate and real-time virtual try-on.
  • Developed and implemented a dynamically adjustable mannequin to simulate various body sizes and poses.
• Implementation Steps:
  • Garment Capture:
    1. Used a movable mannequin to capture target garments' images and depth data under different body shapes and poses.
    2. Post-processed the captured data, including background segmentation and image alignment.
  • Image Synthesis:
    1. Trained an image-to-image translation model using paired data of the measurement suit and target garments.
    2. Achieved image mapping from the measurement suit to the target garment.
  • Real-time Try-on:
    1. Users wear the measurement suit and stand in front of the camera.
    2. The system converts the measurement suit's image into the target garment's try-on effect in real-time.

Research Outcomes

• Specific Results:
  1. Developed a robotic mannequin capable of accurately capturing garments' physical properties.
  2. Proposed the design of a measurement suit that effectively supports high-precision image synthesis for real-time online try-on.
  3. Built a dataset containing 10 target garments, covering various viewpoints, body shapes, and poses.
• Advantages Over Existing Solutions:
  • Capable of handling dynamic interactions (e.g., stretching and grabbing) and accurately simulating garment wrinkles and fit.
  • Generates higher-resolution and more realistic details compared to single-image-based try-on methods.
  • Allows users to try on garments of different sizes and accurately select the size that fits their body shape.
• Experiments and Evaluation Results:
  • User studies showed high ratings for the system's accuracy in displaying garment size and color (4.38/5).
  • Most participants were able to select the right size of clothing using the virtual try-on system.
  • Compared to state-of-the-art try-on methods (e.g., ACGPN), the proposed method outperformed in image clarity, wrinkle detail preservation, and realism.
• Limitations and Future Directions:
  • The method requires a long time for large-scale garment capture, with a single garment capture process taking about 2 hours, limiting scalability.
  • Currently supports only upper-body garments; future work should extend support to more garment types (e.g., long sleeves, dresses).
  • Does not utilize temporal sequence information, making it unable to reproduce accumulated garment deformations in motion sequences (e.g., dynamic wrinkles).
  • The measurement suit design can be further optimized, such as incorporating more complex and information-rich texture patterns.

In summary, this paper's groundbreaking improvements in virtual try-on technology open up new possibilities for enhancing user experiences in online shopping, while providing significant technical and data support for future research.