VINS: Visual Search for Mobile User Interface Design

Literature Title

VINS: Visual Search for Mobile User Interface Design

Literature Information

• Subject Area: Human-Computer Interaction (HCI), Mobile User Interface (UI) Design, Computer Vision
• Keywords: Dataset, Data-Driven Design, User Interface Design, Design Examples, Wireframes, Information Retrieval, Computer Vision, Deep Learning, Object Detection

Research Background and Problem

• Identified Problems or Challenges:
  • Current UI design example search tools often rely on text-based queries, neglecting the visual structure and content of UIs, resulting in low relevance of retrieved design examples.
  • Existing mobile UI retrieval methods (e.g., Swire and others) have limitations in performance and generalizability, with some failing to comprehensively analyze UI content or relying on predefined hierarchies, lacking support for new input images.
• Importance of the Problem:
  • UI design is a critical component of mobile app development, with its visual organization and structure directly impacting the success of an application.
  • Providing effective design example retrieval tools can inspire creativity, improve design efficiency, and support decision-making in design.
• Research Motivation:
  • Current text-based query mechanisms fail to effectively generate examples aligned with design needs, making the search process for design examples cumbersome and uncertain for designers.
  • There is a need to develop a visual search mechanism that deeply understands the hierarchy and structure of UI content to enhance relevance and performance.

Solution

• Proposed Solution:
  • VINS is a visual search framework based on object detection and deep learning, capable of retrieving structurally similar design examples based on input UI images (wireframes or high-fidelity design images).
  • VINS consists of two core modules: UI component detection and image retrieval, which perform semantic segmentation of UI layouts and example retrieval, respectively.
• Innovations:
  1. VINS Dataset: Provides a large-scale mobile UI dataset spanning different design stages (e.g., abstract wireframes, high-fidelity designs), with accurately annotated UI component categories and positions.
  2. Attention Mechanism + Multimodal Embedding: Utilizes an image autoencoder with an attention mechanism and a label encoder to learn joint feature embeddings of UI structure and content, improving the relevance of retrieval results.
• Implementation Steps and Techniques:
  1. Data Collection and Annotation: Extracted and manually annotated approximately 4,800 UI design images, including wireframes and high-fidelity images.
     • The data annotation process used bounding box labels, adhering to strict high-quality digitization standards.
  2. Object Detection Module:
     • Employed the Single Shot MultiBox Detector (SSD) model to locate and classify various components within the UI.
     • Generated semantic layouts for subsequent processing by the retrieval module.
  3. Image Retrieval Module:
     • Introduced an attention mechanism, combining UI content labels and structural hierarchies to train a multimodal embedding model.
     • Used Euclidean distance to evaluate the similarity between the input query image and other images in the design library.
  4. Retrieval System: Provided a set of design examples with similar structure and functionality based on the input image.

Research Outcomes

• Specific Outcomes:
  • Created a high-quality public dataset (VINS dataset) covering 11 types of UI components (e.g., text, buttons, icons).
  • Proposed a retrieval framework based on joint embedding features of images and content, optimizing the relevance and accuracy of retrieval.
• Advantages over Existing Solutions:
  • Compared to the Swire system, VINS considers both the structural hierarchy and content information of UIs, no longer limited to high-level layout information.
  • Overcame the limitations of predefined hierarchies, demonstrating better adaptability to novel UIs.
  • After fine-tuning and performance optimization, retrieval precision (precision@k) improved by 4%-6% compared to baseline models.
• Experimental Evaluation Results:
  • Detection Module: Achieved 76.39% mAP (mean average precision) and 79.02% AUC on the test set.
  • Retrieval Module: Achieved 92.05% precision for Top-1 retrieval results and 86.48% precision for Top-10 retrieval results.
  • Expert Evaluation: Designers found the retrieval results provided by the system highly useful in terms of layout similarity and design inspiration, with positive user feedback.
• Limitations and Future Directions:
  1. Dataset Expansion: The current VINS dataset only covers 11 UI component types, limiting its applicability. Future work should include annotations for more functional components.
  2. Retrieval Improvements:
     • Integrate UI visual features (e.g., fonts and colors) to enhance the model's visual perception capabilities.
     • Adopt tree-like data structures to capture the hierarchical relationships of UI layouts more precisely.
  3. User Research: Conduct more user testing to validate the tool's adaptability in real-world design workflows.
  4. Cross-Domain Expansion: Extend the VINS framework to other visual design domains, such as web pages and advertising posters.

Conclusion

This paper proposes a visual search framework for mobile UI design, VINS, based on object detection and deep learning, significantly improving the performance of existing design retrieval tools and effectively addressing the limitations of keyword-based searches and the lack of layout perspective. In the future, through dataset expansion, technical optimization, and enhanced user research, VINS is expected to play a greater role in promoting data-driven design, inspiring creativity, and improving decision-making efficiency in design.