GenQuery: Supporting Expressive Visual Search with Generative Models

Title of the Paper

GenQuery: Supporting Expressive Visual Search with Generative Models

Paper Information

• Domain: Integration of visual search and generative models in the design field
• Keywords: Visual search, generative models, creativity support, search intent expression, design tools, user study

Research Background and Problem

• Identified Problems and Challenges:

  • Designers often struggle to translate abstract design intentions into specific search keywords during the early design stages.
  • Image search based on existing tools is limited by their evaluation of overall image similarity, making it difficult for users to focus on fine-grained design elements.
  • Current search methods fail to effectively support divergent thinking, making it harder to discover diverse creative ideas.

• Why This Problem is Important:

  • Visual search is a critical component of the early design process. Helping designers express and execute search intentions more efficiently can significantly enhance creativity generation and design exploration quality.

• Research Motivation and Related Work:

  • Existing research primarily focuses on enriching search modes, such as combining natural language and images for search, and proposes methods that emphasize diversity in design recommendations.
  • Generative models have shown potential in personalized creativity support. Text-to-image models can generate high-quality images from simple keywords, while generative language models can expand user intentions.
  • This study explores the potential of generative models as an intermediary layer in visual search.

Solution

• Proposed Method and Solution:

  • The GenQuery system integrates generative models into visual search, offering three core functionalities:
    1. Query Refinement: Expanding vague user queries by suggesting specific search keywords.
    2. Image-Based Image Modification: Selecting a specific region in an image and replacing it with content from a reference image to generate intention-aligned visuals.
    3. Keyword-Based Image Modification: Generating keywords based on the user's search history, modifying images, and creating new search directions.

• Innovations:

  • Integrating generative models as an intermediary layer in the search process, enabling generated images to both express user intentions and serve as starting points for further searches.
  • Leveraging the uncertainty of generative outputs to promote divergent thinking, helping users explore more creative ideas.

• Implementation Steps and Key Technologies:

  • Technical Architecture:
    1. Using GPT-3.5 to generate refined search suggestions.
    2. Employing Segment Anything and PaintByExample models for image region selection and replacement generation.
    3. Utilizing keyword guidance and the Kandinsky2.2 diffusion model for image generation.
  • System Interface: Includes a text input box, image search display, and generate/edit function buttons.
  • Users can interactively preview refinement suggestions or keywords and explore based on the generated results.

Research Outcomes

• Specific Results:

  • Comparative studies show that 16 designers using GenQuery were able to express search intentions more accurately and discover more satisfactory results.
  • Compared to traditional tools, designers using generative features reduced text search frequency by 71.2%, with approximately 35.8% of saved designs obtained through generative search.
  • GenQuery significantly improved the diversity and creativity of results, with high user satisfaction.

• Advantages Over Existing Solutions:

  • Supports users in expressing complex search intentions through image modification and keyword recommendations.
  • Reveals diverse design directions through the generative process, better stimulating divergent exploration compared to traditional search tools.

• Experiments and Evaluation Results:

  • Experiments demonstrated that GenQuery helped users reduce repetitive search behaviors while finding more intention-aligned results through generative search modes.
  • Although generative model outputs occasionally failed to fully match user needs, participants generally benefited from the unexpected exploratory paths they provided.

• Limitations and Future Directions:

  • Limitations:
    1. Generative outputs lack fine-grained control, potentially leading users into repetitive generation loops for specific intentions.
    2. Dataset limitations may result in insufficient search results related to generated images.
    3. The image region selection tool requires further optimization to support more precise user selections.
  • Future Directions:
    1. Adjust the control level of generative models based on the specificity of user intentions.
    2. Integrate user behavior data to improve generative result recommendations.
    3. Expand model support to further explore the application of generative processes in the design prototyping stage.

Conclusion

This paper demonstrates the potential of generative models in enriching visual search tools. By integrating generation and search functionalities, GenQuery supports designers in expressing complex intentions more efficiently during the early creative stages, inspiring more possibilities for creative design. The paper also provides design suggestions for controlling and improving generative model outputs, offering valuable references for the development of future related tools.