FusionProtor: A Mixed-Prototype Tool for Component-level Physical-to-Virtual 3D Transition and Simulation

Research Background and Issues

• Identified Problems and Challenges:
  The authors observed that in traditional 3D prototyping methods, physical and virtual prototypes are often operated independently, leading to fragmented workflows. Additionally, creating high-fidelity prototypes is time-consuming and resource-intensive, which limits designers' creativity. Few design tools effectively utilize information from low-fidelity physical prototypes for detailed modeling.

• Importance of the Problem:
  Concept design determines 70%-80% of a product's lifecycle cost. Efficient prototyping tools are crucial for presenting design information, feasibility testing, and communication with stakeholders.

• Research Motivation and Related Work:
  Emerging technologies such as Generative Artificial Intelligence (GAI) and Extended Reality (XR) offer new solutions for rapid prototype transformation and hybrid interaction. However, their application in 3D prototyping faces limitations, such as the lack of hierarchical component generation capabilities or insufficient utilization of low-fidelity prototypes.

Solution

• Proposed Method or Solution:
  The authors introduced FusionProtor, a hybrid prototyping tool that supports component-level physical-to-virtual 3D transitions and simulations. This tool integrates GAI and XR technologies to enable rapid prototype transitions, refinement, and hybrid interaction.

• Innovations:

  1. Component-Level 3D Creation Method: A novel Component-level Extraction and Generation (ComEG) algorithm is proposed, capable of extracting independent components from designs and generating corresponding high-quality 3D models.
  2. Seamless Physical-to-Virtual Transition: A GAI-based stepwise generation pipeline is provided to quickly transform low-fidelity physical prototypes into high-fidelity virtual designs.
  3. Support for Hybrid Workspaces: XR technology is integrated to design a real-time iterative workflow between physical and virtual environments.

• Implementation Steps and Key Technologies:

  1. Design of Hybrid Workspace: Combines physical and virtual prototypes, allowing designers to interact in real-time using devices like Apple Vision Pro.
  2. Stepwise Transition Pipeline: Establishes a generation pipeline from images to high-fidelity prototypes, including image-to-image transformation and component-separated image-to-3D generation.
  3. Simulation and Animation Modules: Integrates mainstream tools such as Blender and Cinema 4D for component assembly and motion simulation.

Research Outcomes

• Specific Outcomes:
  Developed and validated the FusionProtor tool, which seamlessly connects physical and virtual prototyping workflows, supports rapid transitions from low-fidelity to high-fidelity, and enables component-level iteration and simulation.

• Advantages:

  1. Compared to traditional prototyping tools, FusionProtor significantly improves design efficiency. The average time for designers to complete concept designs is 40.82 minutes, a substantial reduction compared to traditional methods.
  2. Provides robust expressive capabilities, helping designers enrich the details of low-fidelity models and supporting structural logic and modular design.
  3. Supports diverse design tasks, including concept exploration in fields such as smart devices, robotics, and drones.

• Experimental Results:
  In technical evaluations, FusionProtor's component extraction method outperformed mainstream baseline algorithms in terms of extraction quality, image quality, and completion of occluded parts. In user studies, participants rated the tool's usability at 87.81 (SUS score), with a design completion rate of 91.67%.

• Limitations and Future Directions:

  1. GAI's structural understanding remains limited, sometimes resulting in components with incorrect assembly relationships or insufficient geometric accuracy.
  2. Extended use of HMD devices can cause physical discomfort.
  3. Applications in engineering production require further optimization, such as scaling physical prototypes and supporting multi-view inputs.
  4. Early-stage design may face creativity fixation, as premature high-fidelity generation could reduce designers' reflection.

Future work suggestions include optimizing XR interaction features, supporting motion relationship simulation between components, enhancing control over design detail generation, and updating GAI models to accommodate more complex design requirements.