ShapeFindAR: Exploring In-Situ Spatial Search for Physical Artifact Retrieval using Mixed Reality

Document Title

ShapeFindAR: Exploring In-Situ Spatial Search for Physical Artifact Retrieval using Mixed Reality

Document Information

• Domain: Human-Computer Interaction and Personal Fabrication (3D Printing, Mixed Reality)
• Keywords: Personal fabrication, spatial search, in-situ search, mixed reality, physical artifact retrieval, model library, 3D printing

Research Background and Problem Statement

• Identified Challenges:

  • Current 3D printing model retrieval heavily relies on text-based queries, which struggle to effectively convey physical attributes such as scale or shape.
  • The search and preview process is often disconnected from the physical environment, increasing operational complexity.
  • Non-expert users find it difficult to define their needs precisely using text or tags, raising the barriers to using 3D printing and other personal fabrication tools.

• Importance of the Problem:

  • Making personal fabrication more accessible to the general public to meet diverse and personalized needs.
  • Improving the search and preview process to further lower the barriers to personal fabrication, especially for users lacking design skills.

• Research Motivation and Related Work:

  • Existing tools aimed at reducing the barriers to personal fabrication (e.g., simplified modeling interfaces and automated generation tools) still fail to address the direct mapping of physical context to search queries.
  • Preliminary work has explored 3D geometry-based searches and in-situ design tools, but there remains a significant gap in leveraging physical contextual information to enhance the search experience.

Solution

• Methodology and Innovations:

  • Propose a concept of in-situ spatial search combining spatial and text-based queries, allowing users to retrieve 3D printing models directly using their physical environment.
  • Develop ShapeFindAR, a mixed reality prototype application implemented via HoloLens 2 to enable direct in-situ searches.
  • Provide multiple input methods (e.g., spatial sketches, photo tag extraction, text queries) and combined queries, enabling users to express their needs in a relatively imprecise manner.

• Implementation Steps and Techniques:

  1. Spatial Query:
     • Users can draw 3D sketches in the air or generate queries by tracing the contours of real-world objects.
     • A voxel-based and object alignment matching algorithm maps sketch queries to the 3D model database.
  2. Text Query:
     • Supports voice or text-based searches, supplemented by image tag extraction (using Google Cloud Vision API) to improve text query accuracy.
  3. Iterative Search and Result Preview:
     • Users can refine search results by adjusting sketch features of the query or result objects.
     • Conduct in-situ previews in mixed reality to evaluate model compatibility with the environment.
  4. Database and Architecture:
     • Model datasets sourced from Thingiverse and MyMiniFactory.
     • Database built on MongoDB, supporting text indexing and spatial indexing based on geometric features.

Research Outcomes

• Specific Contributions:

  • Introduced the concept of in-situ spatial search and demonstrated its advantages.
  • Developed ShapeFindAR as a prototype implementation of the proposed search concept.
  • Showcased the tool's diverse efficiency and user-friendliness through various use cases (bypassing terminology, iterative optimization, and leveraging contextual features).

• Advantages Compared to Existing Solutions:

  1. More user-friendly, especially for general consumers and novice designers.
  2. Supports multimodal input (text, sketches, image tags), accommodating diverse ways of expressing needs.
  3. Provides more efficient in-situ preview functionality, bridging the gap between design and real-world application environments.

• Experiments and Evaluation:

  • Preliminary testing on a small dataset (4118 objects) showed the system's ability to retrieve relevant objects, though computational efficiency needs improvement in certain search scenarios (e.g., sketch matching).
  • In-situ prototype testing demonstrated its practical significance for non-expert users.

• Limitations and Future Directions:

  • Limitations:
    • Current sketch matching algorithm has low performance, especially with larger datasets.
    • Interpolation algorithms and spatial indexing are not fully compatible with certain sketch inputs, potentially reducing retrieval performance.
    • Sketch-based methods may lack precision for highly specific geometric requirements (e.g., mechanical components).
  • Future Directions:
    • Optimize matching algorithms to improve computational efficiency, such as introducing more advanced geometric matching models.
    • Explore the integration of 3D scanning technology into the search process.
    • Extend the tool to other application domains (e.g., furniture, accessory shopping platforms) and conduct user studies.